2024-02-20T13:25:33.693 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-20T13:25:33.693 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738 --name yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/orig.config.yaml 2024-02-20T13:25:33.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T13:25:33.826 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738 branch: wip-yuri5-testing-2024-02-16-1120-quincy description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7567738' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-02-16-1120-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-02-16-1120-quincy sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4250 sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri5-testing-2024-02-16-1120-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b targets: smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMARbUYPR2E4vAmpXPizT7pEBPn1jeDgO0LZrIFNmAwjmONrV9SoIF1GkMAW8UbBpBfRbZpBlJgYod/GusHIxhk= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCLuTYqzb4pdMWLS7oIsjFu51Zgv8dbXQ91B2L30gC5RWw4KLmGKWcfqRDnaRI7ubMCurPcmqBnaZTQHzbVtJ8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_20:21:18 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-20T13:25:33.826 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa; will attempt to use it 2024-02-20T13:25:33.826 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks 2024-02-20T13:25:33.827 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-20T13:25:33.828 INFO:teuthology.task.internal:Checking packages... 2024-02-20T13:25:33.850 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b' 2024-02-20T13:25:33.850 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:25:33.850 INFO:teuthology.packaging:ref: None 2024-02-20T13:25:33.850 INFO:teuthology.packaging:tag: None 2024-02-20T13:25:33.850 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-02-16-1120-quincy 2024-02-20T13:25:33.850 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:25:33.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-02-16-1120-quincy 2024-02-20T13:25:33.996 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-543.gb0bd01b8 2024-02-20T13:25:33.997 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-20T13:25:34.005 INFO:teuthology.task.internal:no buildpackages task found 2024-02-20T13:25:34.005 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-20T13:25:34.016 INFO:teuthology.task.internal:Saving configuration 2024-02-20T13:25:34.037 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-20T13:25:34.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:25:34.070 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 13:19:59.454176', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMARbUYPR2E4vAmpXPizT7pEBPn1jeDgO0LZrIFNmAwjmONrV9SoIF1GkMAW8UbBpBfRbZpBlJgYod/GusHIxhk='} 2024-02-20T13:25:34.090 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 13:19:59.452132', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCLuTYqzb4pdMWLS7oIsjFu51Zgv8dbXQ91B2L30gC5RWw4KLmGKWcfqRDnaRI7ubMCurPcmqBnaZTQHzbVtJ8='} 2024-02-20T13:25:34.090 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-20T13:25:34.096 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-20T13:25:34.096 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-20T13:25:34.096 INFO:teuthology.run_tasks:Running task console_log... 2024-02-20T13:25:34.165 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fae40ec5160>, signals=[15]) 2024-02-20T13:25:34.166 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-20T13:25:34.172 INFO:teuthology.task.internal:Opening connections... 2024-02-20T13:25:34.173 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2024-02-20T13:25:34.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:25:34.251 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-02-20T13:25:34.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:25:34.326 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-20T13:25:34.333 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2024-02-20T13:25:34.355 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2024-02-20T13:25:34.356 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2024-02-20T13:25:34.417 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2024-02-20T13:25:34.417 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T13:25:34.418 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T13:25:34.419 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2024-02-20T13:25:34.439 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-02-20T13:25:34.461 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-02-20T13:25:34.461 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-02-20T13:25:34.522 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T13:25:34.523 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T13:25:34.524 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-02-20T13:25:34.547 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-20T13:25:34.556 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-20T13:25:34.562 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-20T13:25:34.562 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2024-02-20T13:25:34.565 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-02-20T13:25:34.585 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-20T13:25:34.591 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-20T13:25:34.591 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2024-02-20T13:25:34.626 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-02-20T13:25:34.664 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-20T13:25:34.719 INFO:teuthology.run_tasks:Running task kernel... 2024-02-20T13:25:34.735 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-20T13:25:34.735 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-20T13:25:34.735 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T13:25:34.736 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T13:25:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:25:34.755 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-20T13:25:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:25:34.757 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-02-20T13:25:34.815 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-489.el8.x86_64 2024-02-20T13:25:34.816 INFO:teuthology.task.kernel:Running kernel on smithi158: 4.18.0-489.el8.x86_64 2024-02-20T13:25:34.816 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2024-02-20T13:25:34.818 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-489.el8.x86_64 2024-02-20T13:25:34.818 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-489.el8.x86_64 2024-02-20T13:25:34.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-02-20T13:25:39.747 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - AppStream 12 MB/s | 27 MB 00:02 2024-02-20T13:25:39.782 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 12 MB/s | 27 MB 00:02 2024-02-20T13:25:50.166 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 1.9 MB/s | 10 MB 00:05 2024-02-20T13:25:50.471 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - BaseOS 1.7 MB/s | 10 MB 00:05 2024-02-20T13:25:52.166 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-02-20T13:25:52.575 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-20T13:25:52.613 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2024-02-20T13:25:52.812 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-20T13:25:52.897 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras common packages 40 kB/s | 6.9 kB 00:00 2024-02-20T13:25:53.140 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-20T13:25:53.538 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-02-20T13:25:55.308 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 16 MB 00:02 2024-02-20T13:25:57.304 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 574 kB/s | 24 kB 00:00 2024-02-20T13:25:59.141 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 403 kB/s | 24 kB 00:00 2024-02-20T13:26:00.293 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T13:26:00.293 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T13:26:00.419 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:26:00.419 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout:Install 3 Packages 2024-02-20T13:26:00.420 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:00.421 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 90 M 2024-02-20T13:26:00.421 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 97 M 2024-02-20T13:26:00.421 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:26:02.107 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T13:26:02.107 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T13:26:02.232 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:26:02.232 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:26:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-02-20T13:26:02.232 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout:Install 3 Packages 2024-02-20T13:26:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:02.234 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 90 M 2024-02-20T13:26:02.234 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 97 M 2024-02-20T13:26:02.234 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:26:03.922 INFO:teuthology.orchestra.run.smithi158.stdout:(1/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2024-02-20T13:26:04.306 INFO:teuthology.orchestra.run.smithi158.stdout:(2/3): kernel-4.18.0-539.el8.x86_64.rpm 2.8 MB/s | 10 MB 00:03 2024-02-20T13:26:04.575 INFO:teuthology.orchestra.run.smithi192.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 9.0 MB/s | 10 MB 00:01 2024-02-20T13:26:07.014 INFO:teuthology.orchestra.run.smithi158.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 6.8 MB/s | 43 MB 00:06 2024-02-20T13:26:07.014 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:26:07.014 INFO:teuthology.orchestra.run.smithi158.stdout:Total 14 MB/s | 90 MB 00:06 2024-02-20T13:26:07.592 INFO:teuthology.orchestra.run.smithi192.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 8.7 MB/s | 36 MB 00:04 2024-02-20T13:26:07.697 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:26:08.112 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:26:08.112 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:26:08.759 INFO:teuthology.orchestra.run.smithi192.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 8.1 MB/s | 43 MB 00:05 2024-02-20T13:26:08.759 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:26:08.759 INFO:teuthology.orchestra.run.smithi192.stdout:Total 14 MB/s | 90 MB 00:06 2024-02-20T13:26:09.338 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:26:09.357 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:26:09.440 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:26:09.872 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:26:09.872 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:26:11.130 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:26:11.155 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:26:12.143 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:26:12.622 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:13.915 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:26:14.580 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:16.216 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:16.621 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:18.579 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:20.061 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:20.379 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:22.312 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:42.168 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:42.189 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:42.540 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:42.540 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:42.540 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:43.041 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:43.041 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:43.041 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:26:43.041 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T13:26:43.041 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T13:26:43.042 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:43.042 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:26:43.201 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:26:43.414 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:26:43.414 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:26:43.414 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T13:26:43.415 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T13:26:43.415 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T13:26:43.415 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-20T13:26:43.415 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T13:26:43.415 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:26:43.627 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:26:43.627 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:26:43.628 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2024-02-20T13:26:43.661 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.02 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 129.el8 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Group : System Environment/Base 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 9265586 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T13:26:43.662 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:Relocations : (not relocatable) 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : CentOS BuildSystem 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : CentOS 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:26:43.663 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T13:26:43.664 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T13:26:43.664 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 4.18.0-539.el8.x86_64 2024-02-20T13:26:43.664 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T13:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:44.547 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:44.547 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:26:44.547 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:26:45.017 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:26:45.017 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:45.017 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:26:45.017 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T13:26:45.018 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T13:26:45.018 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:45.018 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:26:45.148 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:26:45.411 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:26:45.411 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:26:45.411 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T13:26:45.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T13:26:45.411 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T13:26:45.411 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-20T13:26:45.411 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T13:26:45.411 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:26:45.480 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2024-02-20T13:26:45.622 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:26:45.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:26:45.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-02-20T13:26:45.655 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-02-20T13:26:45.655 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-02-20T13:26:45.655 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.02 2024-02-20T13:26:45.655 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 129.el8 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Group : System Environment/Base 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 9265586 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Relocations : (not relocatable) 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : CentOS BuildSystem 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T13:26:45.656 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:26:45.657 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T13:26:45.658 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T13:26:45.658 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 4.18.0-539.el8.x86_64 2024-02-20T13:26:45.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T13:26:47.407 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-02-20T13:26:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:done 2024-02-20T13:26:47.604 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-02-20T13:26:47.638 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.eIOy5gxvvd 2024-02-20T13:26:47.639 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eIOy5gxvvd 2024-02-20T13:26:47.712 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.eIOy5gxvvd 2024-02-20T13:26:47.855 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.eIOy5gxvvd is 6KB 2024-02-20T13:26:47.903 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.eIOy5gxvvd 2024-02-20T13:26:47.921 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T13:26:47.986 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T13:26:47.986 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T13:26:47.986 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T13:26:47.986 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T13:26:47.987 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T13:26:48.151 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2024-02-20T13:26:49.729 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-02-20T13:26:49.730 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-02-20T13:26:49.811 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.b7Jx36nQXP 2024-02-20T13:26:49.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.b7Jx36nQXP 2024-02-20T13:26:49.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.b7Jx36nQXP 2024-02-20T13:26:50.035 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.b7Jx36nQXP is 6KB 2024-02-20T13:26:50.083 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.b7Jx36nQXP 2024-02-20T13:26:50.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T13:26:50.169 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T13:26:50.169 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T13:26:50.170 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T13:26:50.170 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T13:26:50.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T13:26:50.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-02-20T13:27:18.182 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-20T13:27:18.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:27:18.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:27:20.336 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-20T13:27:20.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-20T13:27:20.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:27:52.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-20T13:28:01.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:28:01.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:28:04.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-20T13:28:16.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:28:16.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:28:20.340 DEBUG:teuthology.orchestra.remote:timed out 2024-02-20T13:28:29.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-20T13:28:29.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:28:29.670 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-02-20T13:28:29.927 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-20T13:28:29.927 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-20T13:28:29.927 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-02-20T13:28:29.945 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-539.el8.x86_64 2024-02-20T13:28:29.945 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T13:28:29.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T13:28:29.945 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T13:28:30.946 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-20T13:28:30.946 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T13:28:31.006 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-02-20T13:28:31.021 DEBUG:teuthology.parallel:result is None 2024-02-20T13:28:35.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-20T13:28:50.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:28:50.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:28:51.438 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-20T13:28:52.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:28:52.210 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-20T13:28:52.211 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-20T13:28:52.228 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-539.el8.x86_64 2024-02-20T13:28:52.228 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T13:28:52.228 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T13:28:52.228 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T13:28:53.229 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-20T13:28:53.229 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T13:28:53.309 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2024-02-20T13:28:53.324 DEBUG:teuthology.parallel:result is None 2024-02-20T13:28:53.324 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-20T13:28:53.331 INFO:teuthology.task.internal:Creating test directory... 2024-02-20T13:28:53.331 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T13:28:53.333 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T13:28:53.356 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-20T13:28:53.363 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-20T13:28:53.371 INFO:teuthology.task.internal:Creating archive directory... 2024-02-20T13:28:53.372 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T13:28:53.392 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T13:28:53.488 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-20T13:28:53.495 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-20T13:28:53.495 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T13:28:53.497 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T13:28:53.540 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:28:53.542 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:28:53.550 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:28:53.554 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:28:53.556 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-20T13:28:53.563 INFO:teuthology.task.internal:Configuring sudo... 2024-02-20T13:28:53.564 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T13:28:53.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T13:28:53.596 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-20T13:28:53.607 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-20T13:28:53.607 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T13:28:53.635 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T13:28:53.656 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:28:53.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:28:53.799 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:28:53.844 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:28:53.884 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:28:53.885 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T13:28:53.951 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:28:53.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:28:54.053 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:28:54.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:28:54.183 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:28:54.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T13:28:54.250 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2024-02-20T13:28:54.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-02-20T13:28:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:28:54.362 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:28:54.827 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-20T13:28:54.834 INFO:teuthology.task.internal:Starting timer... 2024-02-20T13:28:54.834 INFO:teuthology.run_tasks:Running task pcp... 2024-02-20T13:28:54.846 INFO:teuthology.run_tasks:Running task selinux... 2024-02-20T13:28:54.857 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-20T13:28:54.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2024-02-20T13:28:54.935 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2024-02-20T13:28:54.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-02-20T13:28:55.025 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-02-20T13:28:55.027 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-20T13:28:55.027 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2024-02-20T13:28:55.060 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2024-02-20T13:28:55.060 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-02-20T13:28:55.093 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-02-20T13:28:55.093 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-02-20T13:28:55.094 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:28:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:28:55.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:28:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:28:55.160 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-20T13:28:55.161 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2024-02-20T13:28:55.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-02-20T13:28:55.228 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-20T13:28:55.238 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-20T13:28:55.244 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-20T13:28:55.362 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-20T13:28:55.392 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-20T13:28:55.394 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi158.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T13:35:13.039 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi158.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-02-20T13:35:13.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:35:13.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:35:13.141 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-20T13:35:13.211 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-20T13:35:13.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-20T13:35:13.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:35:13.288 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-02-20T13:35:13.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-20T13:35:13.316 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T13:35:13.326 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T13:35:13.327 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T13:35:13.327 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:35:13.329 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T13:35:13.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:35:13.362 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T13:35:13.376 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T13:35:13.388 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T13:35:13.402 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T13:35:13.446 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2024-02-20T13:35:13.457 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2024-02-20T13:35:13.472 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T13:35:13.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-02-20T13:35:13.483 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-02-20T13:35:13.485 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T13:35:13.497 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T13:35:13.511 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T13:35:13.549 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2024-02-20T13:35:13.552 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:35:13.553 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-20T13:35:13.553 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.553 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.553 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.553 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.587 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:35:13.592 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T13:35:13.603 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T13:35:13.604 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi158.front.sepia.ceph.com 2024-02-20T13:35:13.604 DEBUG:teuthology.orchestra.run.smithi158:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T13:35:13.604 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi192.front.sepia.ceph.com 2024-02-20T13:35:13.604 DEBUG:teuthology.orchestra.run.smithi192:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T13:35:14.392 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:51 ago on Tue 20 Feb 2024 01:33:23 PM UTC. 2024-02-20T13:35:14.417 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:51 ago on Tue 20 Feb 2024 01:33:23 PM UTC. 2024-02-20T13:35:16.413 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout:Resetting modules: 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:16.415 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:16.515 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:Resetting modules: 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:16.516 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:16.673 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:35:16.755 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:35:17.543 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 01:33:23 PM UTC. 2024-02-20T13:35:17.621 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 01:33:23 PM UTC. 2024-02-20T13:35:19.571 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:35:19.607 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:19.607 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout:Installing group/module packages: 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T13:35:19.608 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout:Installing weak dependencies: 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout:Installing module profiles: 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools/common 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout:Enabling module streams: 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools rhel8 2024-02-20T13:35:19.609 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:Install 13 Packages 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 28 M 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 92 M 2024-02-20T13:35:19.610 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:35:19.631 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout:Installing group/module packages: 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T13:35:19.666 INFO:teuthology.orchestra.run.smithi192.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout:Installing module profiles: 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools/common 2024-02-20T13:35:19.667 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools rhel8 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-02-20T13:35:19.668 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:19.669 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 28 M 2024-02-20T13:35:19.669 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 92 M 2024-02-20T13:35:19.669 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:35:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.9 MB/s | 688 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.9 MB/s | 688 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.6 MB/s | 1.0 MB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 565 kB/s | 38 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 3.3 MB/s | 257 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi158.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.4 MB/s | 1.0 MB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi158.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 410 kB/s | 38 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi158.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 3.7 MB/s | 257 kB 00:00 2024-02-20T13:35:20.213 INFO:teuthology.orchestra.run.smithi158.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.6 MB/s | 180 kB 00:00 2024-02-20T13:35:20.240 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.6 MB/s | 180 kB 00:00 2024-02-20T13:35:20.266 INFO:teuthology.orchestra.run.smithi158.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2024-02-20T13:35:20.307 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-02-20T13:35:20.332 INFO:teuthology.orchestra.run.smithi158.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-20T13:35:20.415 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 866 kB/s | 94 kB 00:00 2024-02-20T13:35:20.675 INFO:teuthology.orchestra.run.smithi158.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 7.6 MB/s | 4.3 MB 00:00 2024-02-20T13:35:20.707 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 7.5 MB/s | 4.3 MB 00:00 2024-02-20T13:35:20.925 INFO:teuthology.orchestra.run.smithi158.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 7.6 MB/s | 9.7 MB 00:01 2024-02-20T13:35:20.932 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 7.9 MB/s | 9.7 MB 00:01 2024-02-20T13:35:20.983 INFO:teuthology.orchestra.run.smithi158.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 8.2 MB/s | 2.5 MB 00:00 2024-02-20T13:35:20.999 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 792 kB/s | 53 kB 00:00 2024-02-20T13:35:21.009 INFO:teuthology.orchestra.run.smithi158.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 631 kB/s | 53 kB 00:00 2024-02-20T13:35:21.050 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): toolbox-0.0.99.5-1.module_el8+847+7863 7.4 MB/s | 2.5 MB 00:00 2024-02-20T13:35:21.051 INFO:teuthology.orchestra.run.smithi158.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 613 kB/s | 41 kB 00:00 2024-02-20T13:35:21.075 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 544 kB/s | 41 kB 00:00 2024-02-20T13:35:21.242 INFO:teuthology.orchestra.run.smithi158.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 9.7 MB/s | 8.8 MB 00:00 2024-02-20T13:35:21.243 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:35:21.243 INFO:teuthology.orchestra.run.smithi158.stdout:Total 17 MB/s | 28 MB 00:01 2024-02-20T13:35:21.400 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 8.9 MB/s | 8.8 MB 00:00 2024-02-20T13:35:21.400 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:35:21.400 INFO:teuthology.orchestra.run.smithi192.stdout:Total 16 MB/s | 28 MB 00:01 2024-02-20T13:35:21.511 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:35:21.531 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:35:21.531 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:35:21.669 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:35:21.690 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:35:21.690 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:35:21.800 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:35:21.803 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:35:21.955 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:35:21.958 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:35:22.520 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:35:22.631 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:35:23.456 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T13:35:23.593 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T13:35:23.662 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:35:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:35:23.849 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T13:35:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T13:35:24.058 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T13:35:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T13:35:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T13:35:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T13:35:24.914 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T13:35:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T13:35:25.938 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T13:35:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T13:35:26.202 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T13:35:26.266 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T13:35:26.461 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T13:35:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T13:35:26.922 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T13:35:26.941 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T13:35:27.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:35:27.189 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:35:27.400 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:35:27.411 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:35:27.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:35:27.732 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:35:28.017 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:35:28.018 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T13:35:28.191 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:35:28.192 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:28.544 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:35:28.717 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T13:35:28.717 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T13:35:28.718 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T13:35:28.719 INFO:teuthology.orchestra.run.smithi158.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T13:35:28.719 INFO:teuthology.orchestra.run.smithi158.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T13:35:28.719 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:28.719 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:35:28.817 DEBUG:teuthology.parallel:result is None 2024-02-20T13:35:29.026 DEBUG:teuthology.parallel:result is None 2024-02-20T13:35:29.026 INFO:teuthology.run_tasks:Running task install... 2024-02-20T13:35:29.037 DEBUG:teuthology.task.install:project ceph 2024-02-20T13:35:29.037 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'}} 2024-02-20T13:35:29.037 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'} 2024-02-20T13:35:29.037 INFO:teuthology.task.install:Using flavor: default 2024-02-20T13:35:29.045 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T13:35:29.045 INFO:teuthology.task.install:extra packages: [] 2024-02-20T13:35:29.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b', 'tag': None, 'wait_for_package': False} 2024-02-20T13:35:29.046 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:35:29.046 INFO:teuthology.packaging:ref: None 2024-02-20T13:35:29.046 INFO:teuthology.packaging:tag: None 2024-02-20T13:35:29.046 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:35:29.046 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:35:29.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:35:29.049 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b', 'tag': None, 'wait_for_package': False} 2024-02-20T13:35:29.049 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:35:29.049 INFO:teuthology.packaging:ref: None 2024-02-20T13:35:29.049 INFO:teuthology.packaging:tag: None 2024-02-20T13:35:29.049 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:35:29.049 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:35:29.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:35:29.314 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-20T13:35:29.314 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-20T13:35:29.394 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-20T13:35:29.394 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-20T13:35:29.514 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T13:35:29.515 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:35:29.515 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T13:35:29.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T13:35:29.543 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:35:29.543 INFO:teuthology.packaging:ref: None 2024-02-20T13:35:29.543 INFO:teuthology.packaging:tag: None 2024-02-20T13:35:29.543 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:35:29.543 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:35:29.543 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T13:35:29.574 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T13:35:29.574 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:35:29.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T13:35:29.603 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T13:35:29.603 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:35:29.603 INFO:teuthology.packaging:ref: None 2024-02-20T13:35:29.603 INFO:teuthology.packaging:tag: None 2024-02-20T13:35:29.603 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:35:29.603 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:35:29.604 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T13:35:29.610 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T13:35:29.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T13:35:29.687 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T13:35:29.749 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T13:35:29.755 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2024-02-20T13:35:29.756 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2024-02-20T13:35:29.815 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-02-20T13:35:29.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-02-20T13:35:30.021 INFO:teuthology.orchestra.run.smithi158.stdout:76 files removed 2024-02-20T13:35:30.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw 2024-02-20T13:35:30.080 INFO:teuthology.orchestra.run.smithi192.stdout:76 files removed 2024-02-20T13:35:30.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-02-20T13:35:30.876 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 181 kB/s | 76 kB 00:00 2024-02-20T13:35:30.882 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 208 kB/s | 76 kB 00:00 2024-02-20T13:35:31.226 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 49 kB/s | 15 kB 00:00 2024-02-20T13:35:31.258 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2024-02-20T13:35:31.527 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 5.3 kB/s | 1.4 kB 00:00 2024-02-20T13:35:31.625 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 4.3 kB/s | 1.4 kB 00:00 2024-02-20T13:35:33.586 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 13 MB/s | 27 MB 00:02 2024-02-20T13:35:33.684 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - AppStream 13 MB/s | 27 MB 00:02 2024-02-20T13:35:42.678 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 2.4 MB/s | 10 MB 00:04 2024-02-20T13:35:43.393 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - BaseOS 2.1 MB/s | 10 MB 00:04 2024-02-20T13:35:44.691 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-02-20T13:35:44.962 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 43 kB/s | 6.9 kB 00:00 2024-02-20T13:35:45.393 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-02-20T13:35:45.677 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras common packages 39 kB/s | 6.9 kB 00:00 2024-02-20T13:35:45.789 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - PowerTools 6.6 MB/s | 4.8 MB 00:00 2024-02-20T13:35:46.503 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - PowerTools 6.6 MB/s | 4.8 MB 00:00 2024-02-20T13:35:46.849 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-20T13:35:47.538 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-02-20T13:35:48.057 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-02-20T13:35:48.297 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-02-20T13:35:51.866 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 414 kB/s | 24 kB 00:00 2024-02-20T13:35:52.080 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 551 kB/s | 24 kB 00:00 2024-02-20T13:35:55.395 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:35:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-20T13:35:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout:Install 21 Packages 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 2 Packages 2024-02-20T13:35:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:35:55.401 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 58 M 2024-02-20T13:35:55.401 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:35:55.617 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:35:55.619 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:55.619 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T13:35:55.620 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-20T13:35:55.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:Install 21 Packages 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 58 M 2024-02-20T13:35:55.622 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:35:57.108 INFO:teuthology.orchestra.run.smithi158.stdout:(1/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 3.8 MB/s | 6.1 MB 00:01 2024-02-20T13:35:57.225 INFO:teuthology.orchestra.run.smithi158.stdout:(2/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 220 kB/s | 26 kB 00:00 2024-02-20T13:35:57.701 INFO:teuthology.orchestra.run.smithi158.stdout:(3/23): ceph-common-16.2.14-548.g81bd20d6.el8.x 11 MB/s | 24 MB 00:02 2024-02-20T13:35:57.734 INFO:teuthology.orchestra.run.smithi158.stdout:(4/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 1.6 MB/s | 825 kB 00:00 2024-02-20T13:35:57.793 INFO:teuthology.orchestra.run.smithi158.stdout:(5/23): libradosstriper1-16.2.14-548.g81bd20d6. 5.3 MB/s | 494 kB 00:00 2024-02-20T13:35:57.860 INFO:teuthology.orchestra.run.smithi158.stdout:(6/23): python3-ceph-argparse-16.2.14-548.g81bd 676 kB/s | 45 kB 00:00 2024-02-20T13:35:57.927 INFO:teuthology.orchestra.run.smithi158.stdout:(7/23): python3-ceph-common-16.2.14-548.g81bd20 1.4 MB/s | 99 kB 00:00 2024-02-20T13:35:57.993 INFO:teuthology.orchestra.run.smithi158.stdout:(8/23): python3-cephfs-16.2.14-548.g81bd20d6.el 3.2 MB/s | 215 kB 00:00 2024-02-20T13:35:58.086 INFO:teuthology.orchestra.run.smithi158.stdout:(9/23): python3-rados-16.2.14-548.g81bd20d6.el8 4.1 MB/s | 389 kB 00:00 2024-02-20T13:35:58.148 INFO:teuthology.orchestra.run.smithi192.stdout:(1/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8. 5.4 MB/s | 12 MB 00:02 2024-02-20T13:35:58.153 INFO:teuthology.orchestra.run.smithi158.stdout:(10/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 5.4 MB/s | 369 kB 00:00 2024-02-20T13:35:58.203 INFO:teuthology.orchestra.run.smithi158.stdout:(11/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 2.2 MB/s | 114 kB 00:00 2024-02-20T13:35:58.223 INFO:teuthology.orchestra.run.smithi192.stdout:(2/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 2.7 MB/s | 6.1 MB 00:02 2024-02-20T13:35:58.249 INFO:teuthology.orchestra.run.smithi192.stdout:(3/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 256 kB/s | 26 kB 00:00 2024-02-20T13:35:58.357 INFO:teuthology.orchestra.run.smithi192.stdout:(4/23): libradosstriper1-16.2.14-548.g81bd20d6. 4.5 MB/s | 494 kB 00:00 2024-02-20T13:35:58.386 INFO:teuthology.orchestra.run.smithi158.stdout:(12/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_ 5.8 MB/s | 3.8 MB 00:00 2024-02-20T13:35:58.408 INFO:teuthology.orchestra.run.smithi192.stdout:(5/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 4.4 MB/s | 825 kB 00:00 2024-02-20T13:35:58.467 INFO:teuthology.orchestra.run.smithi192.stdout:(6/23): python3-ceph-argparse-16.2.14-548.g81bd 770 kB/s | 45 kB 00:00 2024-02-20T13:35:58.525 INFO:teuthology.orchestra.run.smithi192.stdout:(7/23): python3-ceph-common-16.2.14-548.g81bd20 1.7 MB/s | 99 kB 00:00 2024-02-20T13:35:58.617 INFO:teuthology.orchestra.run.smithi192.stdout:(8/23): python3-cephfs-16.2.14-548.g81bd20d6.el 2.3 MB/s | 215 kB 00:00 2024-02-20T13:35:58.726 INFO:teuthology.orchestra.run.smithi192.stdout:(9/23): python3-rados-16.2.14-548.g81bd20d6.el8 3.5 MB/s | 389 kB 00:00 2024-02-20T13:35:58.776 INFO:teuthology.orchestra.run.smithi192.stdout:(10/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_ 9.1 MB/s | 3.8 MB 00:00 2024-02-20T13:35:58.788 INFO:teuthology.orchestra.run.smithi158.stdout:(13/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8 3.5 MB/s | 12 MB 00:03 2024-02-20T13:35:58.826 INFO:teuthology.orchestra.run.smithi192.stdout:(11/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 2.2 MB/s | 114 kB 00:00 2024-02-20T13:35:58.851 INFO:teuthology.orchestra.run.smithi192.stdout:(12/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 2.9 MB/s | 369 kB 00:00 2024-02-20T13:35:59.101 INFO:teuthology.orchestra.run.smithi158.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 364 kB/s | 259 kB 00:00 2024-02-20T13:35:59.269 INFO:teuthology.orchestra.run.smithi192.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 623 kB/s | 259 kB 00:00 2024-02-20T13:35:59.321 INFO:teuthology.orchestra.run.smithi158.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 82 kB/s | 44 kB 00:00 2024-02-20T13:35:59.394 INFO:teuthology.orchestra.run.smithi192.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 352 kB/s | 44 kB 00:00 2024-02-20T13:35:59.430 INFO:teuthology.orchestra.run.smithi158.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 552 kB/s | 677 kB 00:01 2024-02-20T13:35:59.528 INFO:teuthology.orchestra.run.smithi192.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-02-20T13:35:59.530 INFO:teuthology.orchestra.run.smithi158.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 110 kB/s | 47 kB 00:00 2024-02-20T13:35:59.619 INFO:teuthology.orchestra.run.smithi192.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 3.3 MB/s | 306 kB 00:00 2024-02-20T13:35:59.670 INFO:teuthology.orchestra.run.smithi192.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 3.6 MB/s | 181 kB 00:00 2024-02-20T13:35:59.695 INFO:teuthology.orchestra.run.smithi192.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 2.3 MB/s | 59 kB 00:00 2024-02-20T13:35:59.753 INFO:teuthology.orchestra.run.smithi192.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-02-20T13:35:59.981 INFO:teuthology.orchestra.run.smithi158.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 133 kB/s | 59 kB 00:00 2024-02-20T13:36:00.023 INFO:teuthology.orchestra.run.smithi158.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 305 kB/s | 181 kB 00:00 2024-02-20T13:36:00.064 INFO:teuthology.orchestra.run.smithi158.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 412 kB/s | 306 kB 00:00 2024-02-20T13:36:00.148 INFO:teuthology.orchestra.run.smithi158.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 449 kB/s | 75 kB 00:00 2024-02-20T13:36:00.337 INFO:teuthology.orchestra.run.smithi192.stdout:(20/23): librdkafka-1.6.1-1.el8.x86_64.rpm 448 kB/s | 677 kB 00:01 2024-02-20T13:36:00.412 INFO:teuthology.orchestra.run.smithi192.stdout:(21/23): librados2-16.2.14-548.g81bd20d6.el8.x8 5.7 MB/s | 3.7 MB 00:00 2024-02-20T13:36:00.507 INFO:teuthology.orchestra.run.smithi158.stdout:(22/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 9.5 MB/s | 4.2 MB 00:00 2024-02-20T13:36:00.715 INFO:teuthology.orchestra.run.smithi158.stdout:(23/23): librados2-16.2.14-548.g81bd20d6.el8.x8 5.4 MB/s | 3.7 MB 00:00 2024-02-20T13:36:00.719 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:00.720 INFO:teuthology.orchestra.run.smithi158.stdout:Total 11 MB/s | 58 MB 00:05 2024-02-20T13:36:00.761 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:36:00.807 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:36:00.807 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:36:00.821 INFO:teuthology.orchestra.run.smithi192.stdout:(22/23): ceph-common-16.2.14-548.g81bd20d6.el8. 5.0 MB/s | 24 MB 00:04 2024-02-20T13:36:00.971 INFO:teuthology.orchestra.run.smithi192.stdout:(23/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 6.6 MB/s | 4.2 MB 00:00 2024-02-20T13:36:00.974 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:00.974 INFO:teuthology.orchestra.run.smithi192.stdout:Total 11 MB/s | 58 MB 00:05 2024-02-20T13:36:01.017 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:36:01.065 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:36:01.065 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:36:01.158 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:36:01.161 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:36:01.423 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:36:01.426 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:36:02.165 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:36:02.417 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-20T13:36:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:36:02.592 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:36:02.645 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-20T13:36:02.809 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:36:02.968 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:36:03.184 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:03.207 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:36:03.281 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:03.417 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:36:03.512 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:03.696 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:03.765 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:36:03.942 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:04.114 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:04.283 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:36:04.345 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:04.345 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:36:04.517 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:36:04.576 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-20T13:36:04.584 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:36:04.777 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-20T13:36:05.031 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-20T13:36:05.200 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:36:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:36:05.305 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-20T13:36:05.456 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-20T13:36:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:36:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:36:05.597 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-20T13:36:05.731 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-20T13:36:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:36:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-20T13:36:05.984 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:36:06.070 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:36:06.175 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-20T13:36:06.309 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:36:06.317 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:36:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:36:06.524 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-20T13:36:06.613 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-20T13:36:06.667 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:36:06.738 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:36:06.800 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-20T13:36:06.977 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-20T13:36:07.002 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-20T13:36:07.158 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-20T13:36:07.183 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-20T13:36:07.378 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-20T13:36:07.592 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-20T13:36:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:10.000 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:10.109 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:10.350 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:10.701 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:11.019 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:36:11.055 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:36:11.131 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:36:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:11.362 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:36:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:36:11.624 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /run 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2024-02-20T13:36:30.536 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2024-02-20T13:36:30.537 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /run 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-02-20T13:36:30.600 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-02-20T13:36:30.601 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-02-20T13:36:30.601 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:36:31.803 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:36:31.803 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:36:31.851 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:36:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:36:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:36:32.030 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:36:32.031 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:32.131 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:36:32.131 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:36:32.722 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-20T13:36:32.723 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-20T13:36:32.724 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:32.724 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-20T13:36:32.724 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-20T13:36:32.834 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:32.834 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-20T13:36:32.834 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-20T13:36:32.835 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-20T13:36:32.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-20T13:36:33.573 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T13:36:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:33.576 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:33.655 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:36:33.906 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-test 2024-02-20T13:36:33.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-02-20T13:36:34.670 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:43 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:36:34.701 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:42 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:36:36.215 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:36:36.216 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:36.216 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:36:36.216 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:36.216 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:36:36.216 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-20T13:36:36.217 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:36.218 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 53 M 2024-02-20T13:36:36.218 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 197 M 2024-02-20T13:36:36.218 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:36:36.264 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:36:36.265 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:36.265 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:36:36.265 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:36.265 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-02-20T13:36:36.266 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:36.267 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 53 M 2024-02-20T13:36:36.267 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 197 M 2024-02-20T13:36:36.267 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:36:36.436 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 985 kB/s | 174 kB 00:00 2024-02-20T13:36:36.678 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 485 kB/s | 203 kB 00:00 2024-02-20T13:36:36.749 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 587 kB/s | 174 kB 00:00 2024-02-20T13:36:36.845 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 460 kB/s | 188 kB 00:00 2024-02-20T13:36:36.891 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 463 kB/s | 203 kB 00:00 2024-02-20T13:36:36.987 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 498 kB/s | 71 kB 00:00 2024-02-20T13:36:37.088 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 789 kB/s | 323 kB 00:00 2024-02-20T13:36:37.158 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 460 kB/s | 188 kB 00:00 2024-02-20T13:36:37.267 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 860 kB/s | 323 kB 00:00 2024-02-20T13:36:37.300 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 498 kB/s | 71 kB 00:00 2024-02-20T13:36:39.926 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 15 MB/s | 52 MB 00:03 2024-02-20T13:36:39.928 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:39.928 INFO:teuthology.orchestra.run.smithi192.stdout:Total 14 MB/s | 53 MB 00:03 2024-02-20T13:36:39.970 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:36:39.990 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:36:39.990 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:36:40.278 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:36:40.281 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:36:40.762 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:36:40.829 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 11 MB/s | 52 MB 00:04 2024-02-20T13:36:40.831 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:40.831 INFO:teuthology.orchestra.run.smithi158.stdout:Total 12 MB/s | 53 MB 00:04 2024-02-20T13:36:40.873 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:36:40.894 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:36:40.894 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:36:40.975 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T13:36:41.181 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T13:36:41.181 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:36:41.184 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:36:41.336 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:36:41.405 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:36:41.600 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T13:36:41.655 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:36:41.752 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:36:41.879 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T13:36:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T13:36:42.177 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:36:42.258 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:36:42.483 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T13:36:42.676 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:36:46.377 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:36:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:36:46.950 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:36:46.950 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-20T13:36:46.950 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-20T13:36:46.950 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T13:36:46.950 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T13:36:46.951 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T13:36:47.276 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-8.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:47.363 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:36:47.549 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:36:47.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-02-20T13:36:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:36:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-20T13:36:47.834 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-20T13:36:47.835 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T13:36:47.835 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T13:36:47.835 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T13:36:48.261 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T13:36:48.261 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:48.261 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:36:48.261 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: jq-1.6-8.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:48.262 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:36:48.350 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:56 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:36:48.518 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph 2024-02-20T13:36:49.276 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:58 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:36:49.914 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-20T13:36:49.918 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T13:36:49.919 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T13:36:49.920 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T13:36:49.921 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T13:36:49.922 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T13:36:49.922 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:49.922 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:36:49.922 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:36:49.922 INFO:teuthology.orchestra.run.smithi192.stdout:Install 32 Packages 2024-02-20T13:36:49.923 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:36:49.923 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 32 M 2024-02-20T13:36:49.924 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 104 M 2024-02-20T13:36:49.924 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:36:50.185 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2024-02-20T13:36:50.335 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 5.7 MB/s | 1.6 MB 00:00 2024-02-20T13:36:50.867 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:36:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T13:36:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T13:36:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:36:50.873 INFO:teuthology.orchestra.run.smithi158.stdout:Install 32 Packages 2024-02-20T13:36:50.874 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:36:50.874 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 32 M 2024-02-20T13:36:50.874 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 104 M 2024-02-20T13:36:50.874 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:36:50.934 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 2.8 MB/s | 2.4 MB 00:00 2024-02-20T13:36:51.053 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): ceph-mgr-modules-core-16.2.14-548.g81bd 2.0 MB/s | 245 kB 00:00 2024-02-20T13:36:51.411 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86_ 17 MB/s | 19 MB 00:01 2024-02-20T13:36:51.436 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2024-02-20T13:36:51.512 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 3.9 MB/s | 5.2 MB 00:01 2024-02-20T13:36:51.537 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2024-02-20T13:36:51.615 INFO:teuthology.orchestra.run.smithi158.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 35 kB/s | 6.4 kB 00:00 2024-02-20T13:36:51.620 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 185 kB/s | 39 kB 00:00 2024-02-20T13:36:51.646 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.1 MB/s | 457 kB 00:00 2024-02-20T13:36:51.688 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-pytz-2017.2-11.el8.noarch.rpm 324 kB/s | 57 kB 00:00 2024-02-20T13:36:51.830 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2024-02-20T13:36:51.880 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 568 kB/s | 109 kB 00:00 2024-02-20T13:36:51.949 INFO:teuthology.orchestra.run.smithi158.stdout:(2/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 3.1 MB/s | 1.6 MB 00:00 2024-02-20T13:36:51.999 INFO:teuthology.orchestra.run.smithi158.stdout:(3/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 4.3 MB/s | 2.4 MB 00:00 2024-02-20T13:36:52.080 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.4 MB/s | 641 kB 00:00 2024-02-20T13:36:52.091 INFO:teuthology.orchestra.run.smithi158.stdout:(4/32): ceph-mgr-modules-core-16.2.14-548.g81bd 2.6 MB/s | 245 kB 00:00 2024-02-20T13:36:52.181 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 124 kB/s | 44 kB 00:00 2024-02-20T13:36:52.314 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 427 kB/s | 185 kB 00:00 2024-02-20T13:36:52.325 INFO:teuthology.orchestra.run.smithi158.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 673 kB/s | 157 kB 00:00 2024-02-20T13:36:52.414 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-02-20T13:36:52.442 INFO:teuthology.orchestra.run.smithi158.stdout:(6/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 6.3 MB/s | 5.2 MB 00:00 2024-02-20T13:36:52.467 INFO:teuthology.orchestra.run.smithi158.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 273 kB/s | 39 kB 00:00 2024-02-20T13:36:52.523 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-20T13:36:52.534 INFO:teuthology.orchestra.run.smithi158.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 856 kB/s | 57 kB 00:00 2024-02-20T13:36:52.548 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-02-20T13:36:52.573 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 350 kB/s | 173 kB 00:00 2024-02-20T13:36:52.632 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2024-02-20T13:36:52.657 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 539 kB/s | 59 kB 00:00 2024-02-20T13:36:52.676 INFO:teuthology.orchestra.run.smithi158.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 442 kB/s | 103 kB 00:00 2024-02-20T13:36:52.701 INFO:teuthology.orchestra.run.smithi158.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.7 MB/s | 457 kB 00:00 2024-02-20T13:36:52.741 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-02-20T13:36:52.758 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-20T13:36:52.783 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-02-20T13:36:52.867 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2024-02-20T13:36:52.884 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-02-20T13:36:52.901 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-02-20T13:36:52.976 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-02-20T13:36:53.009 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 788 kB/s | 85 kB 00:00 2024-02-20T13:36:53.077 INFO:teuthology.orchestra.run.smithi158.stdout:(11/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86 17 MB/s | 19 MB 00:01 2024-02-20T13:36:53.085 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-02-20T13:36:53.102 INFO:teuthology.orchestra.run.smithi158.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 585 kB/s | 249 kB 00:00 2024-02-20T13:36:53.110 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-02-20T13:36:53.112 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:53.112 INFO:teuthology.orchestra.run.smithi192.stdout:Total 10 MB/s | 32 MB 00:03 2024-02-20T13:36:53.127 INFO:teuthology.orchestra.run.smithi158.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-02-20T13:36:53.190 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:36:53.228 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:36:53.228 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:36:53.236 INFO:teuthology.orchestra.run.smithi158.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 688 kB/s | 109 kB 00:00 2024-02-20T13:36:53.345 INFO:teuthology.orchestra.run.smithi158.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 180 kB/s | 44 kB 00:00 2024-02-20T13:36:53.403 INFO:teuthology.orchestra.run.smithi158.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 672 kB/s | 185 kB 00:00 2024-02-20T13:36:53.462 INFO:teuthology.orchestra.run.smithi158.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2024-02-20T13:36:53.495 INFO:teuthology.orchestra.run.smithi158.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-02-20T13:36:53.521 INFO:teuthology.orchestra.run.smithi158.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 301 kB/s | 18 kB 00:00 2024-02-20T13:36:53.531 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:36:53.534 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:36:53.546 INFO:teuthology.orchestra.run.smithi158.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 558 kB/s | 173 kB 00:00 2024-02-20T13:36:53.571 INFO:teuthology.orchestra.run.smithi158.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-02-20T13:36:53.597 INFO:teuthology.orchestra.run.smithi158.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2024-02-20T13:36:53.622 INFO:teuthology.orchestra.run.smithi158.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2024-02-20T13:36:53.647 INFO:teuthology.orchestra.run.smithi158.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-02-20T13:36:53.697 INFO:teuthology.orchestra.run.smithi158.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-20T13:36:53.723 INFO:teuthology.orchestra.run.smithi158.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-02-20T13:36:53.748 INFO:teuthology.orchestra.run.smithi158.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-02-20T13:36:53.774 INFO:teuthology.orchestra.run.smithi158.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-02-20T13:36:53.815 INFO:teuthology.orchestra.run.smithi158.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-02-20T13:36:53.840 INFO:teuthology.orchestra.run.smithi158.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2024-02-20T13:36:53.866 INFO:teuthology.orchestra.run.smithi158.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-02-20T13:36:53.882 INFO:teuthology.orchestra.run.smithi158.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 343 kB/s | 23 kB 00:00 2024-02-20T13:36:53.887 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:36:53.887 INFO:teuthology.orchestra.run.smithi158.stdout:Total 11 MB/s | 32 MB 00:03 2024-02-20T13:36:53.966 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:36:54.005 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:36:54.006 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:36:54.320 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:36:54.324 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:36:54.636 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:36:54.889 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T13:36:55.108 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T13:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T13:36:55.470 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:36:55.665 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T13:36:55.705 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T13:36:55.862 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T13:36:55.932 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T13:36:56.093 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T13:36:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T13:36:56.278 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T13:36:56.430 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T13:36:56.489 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T13:36:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T13:36:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T13:36:56.841 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T13:36:56.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T13:36:57.027 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T13:36:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T13:36:57.213 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T13:36:57.414 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T13:36:57.439 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T13:36:57.627 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T13:36:57.828 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T13:36:57.856 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T13:36:58.102 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T13:36:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T13:36:58.314 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T13:36:58.509 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T13:36:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T13:36:58.785 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T13:36:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:36:59.003 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T13:36:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T13:36:59.282 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T13:36:59.430 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T13:36:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:36:59.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T13:36:59.779 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T13:36:59.816 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T13:37:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T13:37:00.062 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T13:37:00.261 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T13:37:00.267 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T13:37:00.461 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T13:37:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T13:37:00.699 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T13:37:00.796 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T13:37:00.927 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T13:37:01.102 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T13:37:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T13:37:01.477 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T13:37:01.507 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-20T13:37:01.721 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:37:01.826 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T13:37:02.170 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-20T13:37:02.401 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:37:03.235 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:37:03.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:37:03.887 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:37:03.916 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:37:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:37:04.114 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:37:04.315 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:37:04.516 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:37:04.533 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:37:04.559 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:37:04.665 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:37:04.798 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:37:05.015 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:37:05.052 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T13:37:05.053 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T13:37:05.054 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T13:37:05.055 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T13:37:05.055 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T13:37:05.055 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T13:37:05.055 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T13:37:05.214 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:37:05.232 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:37:05.848 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:37:06.277 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:37:06.277 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T13:37:06.278 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T13:37:06.279 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T13:37:06.280 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T13:37:06.280 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T13:37:06.280 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:06.432 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T13:37:06.433 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T13:37:06.434 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:06.435 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:06.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T13:37:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T13:37:07.245 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:07.246 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:07.483 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:15 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:07.529 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-base 2024-02-20T13:37:08.285 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:17 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:08.981 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:37:09.027 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:09.028 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:37:09.028 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:09.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-02-20T13:37:09.790 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:37:09.837 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:09.839 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:37:09.839 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:09.875 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:17 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:09.968 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install cephadm 2024-02-20T13:37:10.722 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:19 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:11.429 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:11.430 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:37:11.430 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 352 k 2024-02-20T13:37:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:37:11.694 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 325 kB/s | 85 kB 00:00 2024-02-20T13:37:11.694 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:11.694 INFO:teuthology.orchestra.run.smithi192.stdout:Total 323 kB/s | 85 kB 00:00 2024-02-20T13:37:11.694 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:37:11.701 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:37:11.701 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:37:11.703 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:37:11.706 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:37:11.749 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:37:12.275 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:12.279 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:12.279 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:12.279 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:37:12.279 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:12.279 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:37:12.280 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:12.281 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 85 k 2024-02-20T13:37:12.281 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 352 k 2024-02-20T13:37:12.281 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:37:12.448 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:12.540 INFO:teuthology.orchestra.run.smithi158.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 330 kB/s | 85 kB 00:00 2024-02-20T13:37:12.540 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:12.540 INFO:teuthology.orchestra.run.smithi158.stdout:Total 328 kB/s | 85 kB 00:00 2024-02-20T13:37:12.541 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:37:12.547 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:37:12.547 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:37:12.549 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:37:12.552 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:37:12.597 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:37:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:13.068 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:13.086 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T13:37:13.279 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.504 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:37:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:13.894 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:37:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:13.895 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:14.041 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:22 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:14.138 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T13:37:14.898 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:15.586 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:15.587 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:15.588 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 161 k 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 470 k 2024-02-20T13:37:15.589 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:37:15.916 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 492 kB/s | 161 kB 00:00 2024-02-20T13:37:15.916 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:15.917 INFO:teuthology.orchestra.run.smithi192.stdout:Total 490 kB/s | 161 kB 00:00 2024-02-20T13:37:15.917 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:37:15.927 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:37:15.927 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:37:15.974 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:37:15.977 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:37:16.477 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:16.478 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:16.478 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2024-02-20T13:37:16.478 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:37:16.479 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:16.480 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 161 k 2024-02-20T13:37:16.480 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 470 k 2024-02-20T13:37:16.480 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:37:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:37:17.311 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 194 kB/s | 161 kB 00:00 2024-02-20T13:37:17.312 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:17.312 INFO:teuthology.orchestra.run.smithi158.stdout:Total 193 kB/s | 161 kB 00:00 2024-02-20T13:37:17.312 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:37:17.324 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:37:17.325 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:37:17.378 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:37:17.381 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:37:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:17.935 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:37:18.178 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:18.313 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:18.616 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:18.697 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:18.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-02-20T13:37:19.008 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:37:19.008 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:19.009 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:37:19.009 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:37:19.009 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:19.009 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:19.238 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr 2024-02-20T13:37:19.684 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:27 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:20.003 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:29 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:21.190 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:37:21.237 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:21.238 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:37:21.238 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:21.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T13:37:21.496 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:37:21.544 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:21.545 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:37:21.546 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:21.674 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T13:37:22.093 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:22.432 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:31 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:23.652 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.4 M 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 22 M 2024-02-20T13:37:23.654 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:37:23.860 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-grafana-dashboards-16.2.14-548.g81b 174 kB/s | 24 kB 00:00 2024-02-20T13:37:23.968 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-prometheus-alerts-16.2.14-548.g81bd 60 kB/s | 15 kB 00:00 2024-02-20T13:37:23.977 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:23.978 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:23.978 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:37:23.978 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-20T13:37:23.979 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:23.980 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.4 M 2024-02-20T13:37:23.980 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 22 M 2024-02-20T13:37:23.980 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:37:24.052 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 9.4 MB/s | 3.1 MB 00:00 2024-02-20T13:37:24.069 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-02-20T13:37:24.152 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2024-02-20T13:37:24.252 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 981 kB/s | 196 kB 00:00 2024-02-20T13:37:24.255 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:24.255 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.7 MB/s | 3.4 MB 00:00 2024-02-20T13:37:24.265 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:37:24.275 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:37:24.275 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:37:24.421 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:37:24.424 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:37:24.672 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): ceph-grafana-dashboards-16.2.14-548.g81b 164 kB/s | 24 kB 00:00 2024-02-20T13:37:24.700 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:37:24.889 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): ceph-prometheus-alerts-16.2.14-548.g81bd 41 kB/s | 15 kB 00:00 2024-02-20T13:37:24.902 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T13:37:24.914 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 177 kB/s | 43 kB 00:00 2024-02-20T13:37:25.172 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T13:37:25.289 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 84 kB/s | 34 kB 00:00 2024-02-20T13:37:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T13:37:25.456 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 362 kB/s | 196 kB 00:00 2024-02-20T13:37:25.485 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-20T13:37:25.632 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 2.8 MB/s | 3.1 MB 00:01 2024-02-20T13:37:25.635 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:25.635 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.1 MB/s | 3.4 MB 00:01 2024-02-20T13:37:25.647 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:37:25.657 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:37:25.657 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:37:25.827 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:37:25.831 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:37:26.115 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:37:26.340 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T13:37:26.539 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-20T13:37:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T13:37:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:37:26.812 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T13:37:26.933 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-20T13:37:27.105 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:37:27.105 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-20T13:37:27.106 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-20T13:37:27.106 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-20T13:37:27.106 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T13:37:27.106 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T13:37:27.493 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T13:37:27.494 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T13:37:27.494 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:27.494 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:27.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T13:37:27.979 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-20T13:37:28.220 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:37:28.408 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:36 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:37:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:37:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-20T13:37:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-20T13:37:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-20T13:37:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T13:37:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:28.929 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:29.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T13:37:29.925 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:38 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:29.930 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:37:29.931 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T13:37:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout:Install 8 Packages 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 37 M 2024-02-20T13:37:29.933 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 210 M 2024-02-20T13:37:29.934 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:37:31.479 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T13:37:31.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout:Install 8 Packages 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:31.482 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 37 M 2024-02-20T13:37:31.483 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 210 M 2024-02-20T13:37:31.483 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:37:31.808 INFO:teuthology.orchestra.run.smithi192.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-54 7.2 MB/s | 8.6 MB 00:01 2024-02-20T13:37:33.262 INFO:teuthology.orchestra.run.smithi158.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-54 6.1 MB/s | 8.6 MB 00:01 2024-02-20T13:37:37.871 INFO:teuthology.orchestra.run.smithi158.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 829 kB/s | 3.7 MB 00:04 2024-02-20T13:37:38.059 INFO:teuthology.orchestra.run.smithi192.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 611 kB/s | 3.7 MB 00:06 2024-02-20T13:37:38.146 INFO:teuthology.orchestra.run.smithi158.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 819 kB/s | 225 kB 00:00 2024-02-20T13:37:38.367 INFO:teuthology.orchestra.run.smithi192.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 731 kB/s | 225 kB 00:00 2024-02-20T13:37:38.518 INFO:teuthology.orchestra.run.smithi192.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 629 kB/s | 4.9 MB 00:07 2024-02-20T13:37:38.760 INFO:teuthology.orchestra.run.smithi192.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.6 MB/s | 645 kB 00:00 2024-02-20T13:37:39.093 INFO:teuthology.orchestra.run.smithi192.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 518 kB/s | 172 kB 00:00 2024-02-20T13:37:39.305 INFO:teuthology.orchestra.run.smithi158.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 649 kB/s | 4.7 MB 00:07 2024-02-20T13:37:39.513 INFO:teuthology.orchestra.run.smithi158.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 649 kB/s | 4.9 MB 00:07 2024-02-20T13:37:39.644 INFO:teuthology.orchestra.run.smithi192.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 536 kB/s | 4.7 MB 00:09 2024-02-20T13:37:39.764 INFO:teuthology.orchestra.run.smithi158.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 690 kB/s | 172 kB 00:00 2024-02-20T13:37:39.989 INFO:teuthology.orchestra.run.smithi158.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 943 kB/s | 645 kB 00:00 2024-02-20T13:37:47.305 INFO:teuthology.orchestra.run.smithi158.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.5 MB/s | 14 MB 00:09 2024-02-20T13:37:47.306 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:47.306 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.3 MB/s | 37 MB 00:15 2024-02-20T13:37:47.538 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:37:47.556 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:37:47.556 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:37:47.892 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:37:47.895 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:37:48.457 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:37:48.581 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:37:48.710 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:37:48.864 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:37:49.462 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:37:49.614 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:50.264 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:50.430 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:37:51.051 INFO:teuthology.orchestra.run.smithi192.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.1 MB/s | 14 MB 00:12 2024-02-20T13:37:51.051 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:37:51.051 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.8 MB/s | 37 MB 00:21 2024-02-20T13:37:51.110 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:37:51.283 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:37:51.301 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:37:51.302 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:37:51.426 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:37:51.630 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:37:51.643 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:37:51.647 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:37:52.238 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:37:52.366 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:37:52.506 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:37:52.674 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:37:53.271 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:37:53.402 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:37:53.415 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:54.040 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:54.182 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:37:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T13:37:54.862 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:37:54.896 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:37:55.219 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:37:55.260 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T13:37:55.261 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T13:37:55.465 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:37:55.746 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T13:37:55.746 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:55.746 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:55.747 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:37:55.893 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-rook 2024-02-20T13:37:56.652 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:05 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:37:57.227 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:37:58.228 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:37:58.230 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:37:58.231 INFO:teuthology.orchestra.run.smithi158.stdout:Install 13 Packages 2024-02-20T13:37:58.232 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:37:58.232 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 1.9 M 2024-02-20T13:37:58.232 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 24 M 2024-02-20T13:37:58.232 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:37:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T13:37:58.748 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:37:58.910 INFO:teuthology.orchestra.run.smithi158.stdout:(1/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 185 kB/s | 27 kB 00:00 2024-02-20T13:37:58.935 INFO:teuthology.orchestra.run.smithi158.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 153 kB/s | 27 kB 00:00 2024-02-20T13:37:58.960 INFO:teuthology.orchestra.run.smithi158.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 100 kB/s | 20 kB 00:00 2024-02-20T13:37:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:37:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T13:37:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T13:37:59.219 INFO:teuthology.orchestra.run.smithi158.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 409 kB/s | 126 kB 00:00 2024-02-20T13:37:59.244 INFO:teuthology.orchestra.run.smithi158.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 388 kB/s | 110 kB 00:00 2024-02-20T13:37:59.420 INFO:teuthology.orchestra.run.smithi158.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:37:59.470 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:37:59.487 INFO:teuthology.orchestra.run.smithi158.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 245 kB/s | 16 kB 00:00 2024-02-20T13:37:59.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-02-20T13:37:59.637 INFO:teuthology.orchestra.run.smithi158.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 542 kB/s | 81 kB 00:00 2024-02-20T13:37:59.887 INFO:teuthology.orchestra.run.smithi158.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 64 kB/s | 43 kB 00:00 2024-02-20T13:37:59.912 INFO:teuthology.orchestra.run.smithi158.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 163 kB/s | 155 kB 00:00 2024-02-20T13:38:00.297 INFO:teuthology.orchestra.run.smithi158.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 157 kB/s | 64 kB 00:00 2024-02-20T13:38:00.377 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:08 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:00.424 INFO:teuthology.orchestra.run.smithi158.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2024-02-20T13:38:00.449 INFO:teuthology.orchestra.run.smithi158.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 114 kB/s | 61 kB 00:00 2024-02-20T13:38:00.450 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:00.450 INFO:teuthology.orchestra.run.smithi158.stdout:Total 882 kB/s | 1.9 MB 00:02 2024-02-20T13:38:00.508 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:00.524 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:00.524 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:00.611 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:00.614 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:01.251 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:01.464 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T13:38:01.692 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T13:38:01.907 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T13:38:01.909 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:01.910 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-02-20T13:38:01.911 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.9 M 2024-02-20T13:38:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2024-02-20T13:38:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:01.930 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T13:38:02.143 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T13:38:02.376 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T13:38:02.625 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T13:38:02.892 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T13:38:03.162 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T13:38:04.218 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T13:38:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T13:38:04.525 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 152 kB/s | 20 kB 00:00 2024-02-20T13:38:04.541 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 179 kB/s | 27 kB 00:00 2024-02-20T13:38:04.668 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T13:38:04.683 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 799 kB/s | 126 kB 00:00 2024-02-20T13:38:04.709 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 658 kB/s | 110 kB 00:00 2024-02-20T13:38:04.751 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 77 kB/s | 27 kB 00:00 2024-02-20T13:38:04.809 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 565 kB/s | 33 kB 00:00 2024-02-20T13:38:04.843 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 490 kB/s | 16 kB 00:00 2024-02-20T13:38:04.843 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T13:38:04.893 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.6 MB/s | 81 kB 00:00 2024-02-20T13:38:04.969 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 16 MB/s | 1.2 MB 00:00 2024-02-20T13:38:04.994 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 150 kB/s | 43 kB 00:00 2024-02-20T13:38:05.019 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:38:05.049 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 798 kB/s | 64 kB 00:00 2024-02-20T13:38:05.192 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 305 kB/s | 155 kB 00:00 2024-02-20T13:38:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:38:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T13:38:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T13:38:05.386 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 156 kB/s | 61 kB 00:00 2024-02-20T13:38:05.386 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:05.387 INFO:teuthology.orchestra.run.smithi192.stdout:Total 563 kB/s | 1.9 MB 00:03 2024-02-20T13:38:05.445 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:05.461 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:05.461 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:05.548 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:05.551 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T13:38:05.761 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:05.762 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:05.878 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T13:38:06.140 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T13:38:06.597 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T13:38:06.639 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:15 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:06.849 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T13:38:07.078 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T13:38:07.337 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T13:38:07.596 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T13:38:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T13:38:08.145 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T13:38:08.207 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:08.208 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:38:08.208 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:08.208 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T13:38:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T13:38:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:08.210 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:08.210 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:08.210 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-20T13:38:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:08.211 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 5.9 M 2024-02-20T13:38:08.211 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 24 M 2024-02-20T13:38:08.212 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:08.623 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 393 kB/s | 116 kB 00:00 2024-02-20T13:38:08.648 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.8 MB/s | 578 kB 00:00 2024-02-20T13:38:08.848 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 94 kB/s | 21 kB 00:00 2024-02-20T13:38:09.015 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.0 MB/s | 4.8 MB 00:00 2024-02-20T13:38:09.040 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2024-02-20T13:38:09.082 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 819 kB/s | 355 kB 00:00 2024-02-20T13:38:09.083 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:09.083 INFO:teuthology.orchestra.run.smithi158.stdout:Total 6.7 MB/s | 5.9 MB 00:00 2024-02-20T13:38:09.134 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:09.153 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:09.153 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T13:38:09.247 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:09.250 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:09.461 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T13:38:09.531 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T13:38:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T13:38:09.878 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T13:38:10.030 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T13:38:10.038 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:38:10.336 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T13:38:10.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T13:38:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:10.818 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:10.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T13:38:10.980 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T13:38:11.318 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T13:38:11.537 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T13:38:11.695 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:19 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:11.751 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T13:38:12.025 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-20T13:38:12.417 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-20T13:38:12.418 INFO:teuthology.orchestra.run.smithi158.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-20T13:38:12.418 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-20T13:38:12.418 INFO:teuthology.orchestra.run.smithi158.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-20T13:38:12.418 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:12.418 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:12.562 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-fuse 2024-02-20T13:38:13.234 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T13:38:13.235 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-02-20T13:38:13.236 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:13.237 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.9 M 2024-02-20T13:38:13.237 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2024-02-20T13:38:13.237 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:13.333 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:22 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:13.710 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 376 kB/s | 116 kB 00:00 2024-02-20T13:38:13.735 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.7 MB/s | 578 kB 00:00 2024-02-20T13:38:13.786 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 281 kB/s | 21 kB 00:00 2024-02-20T13:38:13.827 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.8 MB/s | 355 kB 00:00 2024-02-20T13:38:13.853 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 730 kB/s | 49 kB 00:00 2024-02-20T13:38:14.061 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2024-02-20T13:38:14.062 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:14.062 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.1 MB/s | 5.9 MB 00:00 2024-02-20T13:38:14.112 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:14.121 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:14.122 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:14.226 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:14.228 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:14.534 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:14.794 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T13:38:14.887 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:14.888 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout:Install 2 Packages 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 994 k 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 2.8 M 2024-02-20T13:38:14.889 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:15.115 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T13:38:15.166 INFO:teuthology.orchestra.run.smithi158.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 387 kB/s | 83 kB 00:00 2024-02-20T13:38:15.241 INFO:teuthology.orchestra.run.smithi158.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 3.1 MB/s | 911 kB 00:00 2024-02-20T13:38:15.242 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:15.242 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.8 MB/s | 994 kB 00:00 2024-02-20T13:38:15.252 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:15.272 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:15.272 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:15.360 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:15.363 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:15.774 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T13:38:16.001 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:38:16.027 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T13:38:16.382 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:38:16.382 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-20T13:38:16.391 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T13:38:16.650 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T13:38:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T13:38:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:16.768 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T13:38:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:16.768 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:16.836 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:38:17.008 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados-devel 2024-02-20T13:38:17.125 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:38:17.125 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-20T13:38:17.125 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T13:38:17.125 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T13:38:17.125 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T13:38:17.126 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T13:38:17.524 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:17.525 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:17.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-02-20T13:38:17.779 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:26 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:18.424 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:26 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:19.334 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:19.335 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 131 k 2024-02-20T13:38:19.336 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 432 k 2024-02-20T13:38:19.336 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:19.520 INFO:teuthology.orchestra.run.smithi158.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 711 kB/s | 131 kB 00:00 2024-02-20T13:38:19.521 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:19.521 INFO:teuthology.orchestra.run.smithi158.stdout:Total 706 kB/s | 131 kB 00:00 2024-02-20T13:38:19.521 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:19.526 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:19.526 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:19.576 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:19.579 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:19.685 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:19.899 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:19.978 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T13:38:19.979 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:19.980 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:19.980 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:19.980 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-02-20T13:38:19.980 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:19.980 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 994 k 2024-02-20T13:38:19.981 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-02-20T13:38:19.981 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:20.162 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 591 kB/s | 83 kB 00:00 2024-02-20T13:38:20.193 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:20.454 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 2.1 MB/s | 911 kB 00:00 2024-02-20T13:38:20.454 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:20.454 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.0 MB/s | 994 kB 00:00 2024-02-20T13:38:20.464 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:20.484 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:20.484 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:20.549 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:20.549 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:20.571 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:20.574 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:20.831 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs2 2024-02-20T13:38:21.001 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T13:38:21.203 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:38:21.594 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:30 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:21.599 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:38:21.599 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-20T13:38:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T13:38:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:22.016 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T13:38:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:22.016 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:22.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-02-20T13:38:23.018 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:31 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:23.075 INFO:teuthology.orchestra.run.smithi158.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:23.122 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:23.123 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:23.123 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:23.222 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs-devel 2024-02-20T13:38:23.985 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:32 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:24.556 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 131 k 2024-02-20T13:38:24.557 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 432 k 2024-02-20T13:38:24.558 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:24.741 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 714 kB/s | 131 kB 00:00 2024-02-20T13:38:24.742 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:24.742 INFO:teuthology.orchestra.run.smithi192.stdout:Total 710 kB/s | 131 kB 00:00 2024-02-20T13:38:24.742 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:24.747 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:24.747 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:24.796 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:24.799 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:24.905 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:25.142 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:25.471 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:25.537 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:25.537 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:25.537 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2024-02-20T13:38:25.537 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:25.537 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:25.538 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 25 k 2024-02-20T13:38:25.539 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 107 k 2024-02-20T13:38:25.539 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:25.759 INFO:teuthology.orchestra.run.smithi158.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 116 kB/s | 25 kB 00:00 2024-02-20T13:38:25.760 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:25.760 INFO:teuthology.orchestra.run.smithi158.stdout:Total 115 kB/s | 25 kB 00:00 2024-02-20T13:38:25.760 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:25.764 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:25.764 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:25.766 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:25.769 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:25.819 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:25.861 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:26.006 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:26.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-02-20T13:38:26.275 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:26.623 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:26.744 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados2 2024-02-20T13:38:26.881 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:34 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:27.504 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:36 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:28.364 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:28.410 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:28.412 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:28.412 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:28.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-02-20T13:38:29.023 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:29.069 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:29.071 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:29.071 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:29.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librbd1 2024-02-20T13:38:29.264 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:37 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:30.022 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:39 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:30.831 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:30.832 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 25 k 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 107 k 2024-02-20T13:38:30.833 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:30.982 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 172 kB/s | 25 kB 00:00 2024-02-20T13:38:30.982 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:30.982 INFO:teuthology.orchestra.run.smithi192.stdout:Total 170 kB/s | 25 kB 00:00 2024-02-20T13:38:30.982 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:30.986 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:30.986 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:30.988 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:30.991 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:31.039 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:31.516 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:31.547 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:31.593 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:31.595 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:31.595 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:31.709 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rados 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:31.864 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:31.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-02-20T13:38:32.463 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:41 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:32.733 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:40 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:33.959 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:34.005 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:34.006 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:34.006 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:34.251 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:34.288 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rgw 2024-02-20T13:38:34.297 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:34.299 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:34.299 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:34.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-02-20T13:38:35.046 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:44 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:35.157 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:43 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:36.595 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:36.641 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:36.643 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:36.643 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:36.663 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:36.710 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:36.711 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:36.711 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:36.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-cephfs 2024-02-20T13:38:36.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-02-20T13:38:37.516 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:46 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:37.562 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:45 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:39.014 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:39.060 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:39.061 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:39.062 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:39.063 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:39.110 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:39.111 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:39.111 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:39.193 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rbd 2024-02-20T13:38:39.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-02-20T13:38:39.946 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:48 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:40.162 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:48 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:41.449 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:41.495 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:41.496 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-20T13:38:41.496 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:41.606 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-fuse 2024-02-20T13:38:41.677 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:41.724 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:41.725 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:41.725 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:41.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-02-20T13:38:42.366 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:51 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:42.572 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:50 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:43.907 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:43.908 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:43.909 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:38:43.909 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:43.909 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 88 k 2024-02-20T13:38:43.909 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 240 k 2024-02-20T13:38:43.909 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:44.073 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 539 kB/s | 88 kB 00:00 2024-02-20T13:38:44.073 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:44.074 INFO:teuthology.orchestra.run.smithi158.stdout:Total 535 kB/s | 88 kB 00:00 2024-02-20T13:38:44.074 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:44.079 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:44.084 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:44.084 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:44.126 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:44.127 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:44.127 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:44.136 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:44.139 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:44.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-02-20T13:38:44.244 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:44.428 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:44.724 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:44.980 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:52 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:45.285 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:45.285 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:45.286 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:45.286 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:45.286 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:45.286 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:45.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-mirror 2024-02-20T13:38:46.255 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:55 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:46.473 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:38:46.519 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:46.520 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-02-20T13:38:46.520 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:46.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-02-20T13:38:47.372 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:55 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:47.804 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:38:47.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:47.806 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.7 M 2024-02-20T13:38:47.806 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 15 M 2024-02-20T13:38:47.806 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:48.160 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2024-02-20T13:38:48.161 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:48.161 INFO:teuthology.orchestra.run.smithi158.stdout:Total 10 MB/s | 3.7 MB 00:00 2024-02-20T13:38:48.161 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:48.173 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:48.174 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:48.236 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:48.239 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:48.693 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:48.882 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:48.911 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:48.911 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:38:48.912 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:48.913 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 88 k 2024-02-20T13:38:48.913 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 240 k 2024-02-20T13:38:48.913 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:49.186 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 324 kB/s | 88 kB 00:00 2024-02-20T13:38:49.187 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:49.187 INFO:teuthology.orchestra.run.smithi192.stdout:Total 322 kB/s | 88 kB 00:00 2024-02-20T13:38:49.187 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:49.196 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:49.196 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:49.245 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:49.248 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:49.363 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:49.564 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:49.673 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:49.673 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:49.673 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:49.673 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:49.673 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:49.674 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:49.903 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-nbd 2024-02-20T13:38:50.137 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:50.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:50.493 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:50.493 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:50.494 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:50.494 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:50.494 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:50.671 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:59 ago on Tue 20 Feb 2024 01:35:51 PM UTC. 2024-02-20T13:38:50.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-02-20T13:38:51.493 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:59 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:52.227 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-20T13:38:52.227 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:52.227 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-20T13:38:52.227 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:52.227 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:52.228 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 175 k 2024-02-20T13:38:52.229 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 523 k 2024-02-20T13:38:52.229 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-20T13:38:52.416 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 948 kB/s | 175 kB 00:00 2024-02-20T13:38:52.416 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:52.416 INFO:teuthology.orchestra.run.smithi158.stdout:Total 941 kB/s | 175 kB 00:00 2024-02-20T13:38:52.416 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-20T13:38:52.425 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-20T13:38:52.425 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-20T13:38:52.475 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-20T13:38:52.478 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-20T13:38:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-20T13:38:52.843 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:53.052 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:38:53.053 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:53.054 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.7 M 2024-02-20T13:38:53.054 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 15 M 2024-02-20T13:38:53.054 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:53.124 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:53.487 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-20T13:38:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:38:53.487 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-20T13:38:53.615 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 6.5 MB/s | 3.7 MB 00:00 2024-02-20T13:38:53.615 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:53.616 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.5 MB/s | 3.7 MB 00:00 2024-02-20T13:38:53.616 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:53.627 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:53.627 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:53.690 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:53.694 DEBUG:teuthology.parallel:result is None 2024-02-20T13:38:53.695 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:54.127 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:54.381 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:54.793 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:55.201 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:55.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-02-20T13:38:56.213 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:04 ago on Tue 20 Feb 2024 01:35:52 PM UTC. 2024-02-20T13:38:57.773 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-20T13:38:57.773 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:57.773 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-02-20T13:38:57.773 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:57.773 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:57.774 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 175 k 2024-02-20T13:38:57.775 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 523 k 2024-02-20T13:38:57.775 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-20T13:38:58.369 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 295 kB/s | 175 kB 00:00 2024-02-20T13:38:58.369 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:38:58.369 INFO:teuthology.orchestra.run.smithi192.stdout:Total 294 kB/s | 175 kB 00:00 2024-02-20T13:38:58.370 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-20T13:38:58.381 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-20T13:38:58.381 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-20T13:38:58.433 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-20T13:38:58.436 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-20T13:38:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-20T13:38:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:38:59.478 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-20T13:38:59.663 DEBUG:teuthology.parallel:result is None 2024-02-20T13:38:59.664 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:38:59.664 INFO:teuthology.packaging:ref: None 2024-02-20T13:38:59.664 INFO:teuthology.packaging:tag: None 2024-02-20T13:38:59.664 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:38:59.664 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:38:59.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:38:59.916 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T13:38:59.944 INFO:teuthology.orchestra.run.smithi158.stdout:16.2.14-548.g81bd20d6.el8 2024-02-20T13:38:59.944 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-20T13:38:59.944 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-20T13:38:59.946 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:38:59.946 INFO:teuthology.packaging:ref: None 2024-02-20T13:38:59.946 INFO:teuthology.packaging:tag: None 2024-02-20T13:38:59.946 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:38:59.946 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:38:59.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:39:00.201 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T13:39:00.225 INFO:teuthology.orchestra.run.smithi192.stdout:16.2.14-548.g81bd20d6.el8 2024-02-20T13:39:00.226 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-20T13:39:00.226 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-20T13:39:00.227 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T13:39:00.228 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:39:00.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T13:39:00.262 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:39:00.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T13:39:00.295 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T13:39:00.296 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:39:00.296 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T13:39:00.333 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T13:39:00.402 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:39:00.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T13:39:00.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T13:39:00.500 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T13:39:00.500 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:39:00.500 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T13:39:00.529 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T13:39:00.598 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:39:00.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T13:39:00.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T13:39:00.693 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T13:39:00.694 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:39:00.694 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T13:39:00.721 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T13:39:00.788 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:39:00.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T13:39:00.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T13:39:00.883 INFO:teuthology.run_tasks:Running task print... 2024-02-20T13:39:00.894 INFO:teuthology.task.print:**** done install task... 2024-02-20T13:39:00.894 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T13:39:00.994 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'} 2024-02-20T13:39:00.994 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2024-02-20T13:39:00.995 INFO:tasks.cephadm:Cluster fsid is 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:39:00.995 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T13:39:00.995 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T13:39:00.995 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi158': '172.21.15.158', 'mon.smithi192': '172.21.15.192'} 2024-02-20T13:39:00.995 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T13:39:00.995 DEBUG:teuthology.orchestra.run.smithi158:> sudo hostname $(hostname -s) 2024-02-20T13:39:01.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-02-20T13:39:01.050 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2024-02-20T13:39:01.050 DEBUG:teuthology.orchestra.run.smithi158:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.361 INFO:teuthology.orchestra.run.smithi158.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 20 13:39 /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.362 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.551 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 20 13:39 /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.552 DEBUG:teuthology.orchestra.run.smithi158:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.571 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T13:39:01.629 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2024-02-20T13:39:01.629 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-02-20T13:39:01.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-02-20T13:39:02.791 INFO:teuthology.orchestra.run.smithi158.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:39:02.829 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:39:27.877 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout: "repo_digests": [ 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2024-02-20T13:39:27.878 INFO:teuthology.orchestra.run.smithi158.stdout:} 2024-02-20T13:39:36.873 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-02-20T13:39:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-02-20T13:39:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-02-20T13:39:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-02-20T13:39:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-02-20T13:39:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-02-20T13:39:36.874 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-02-20T13:39:36.897 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph 2024-02-20T13:39:36.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-02-20T13:39:36.966 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /etc/ceph 2024-02-20T13:39:37.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-02-20T13:39:37.035 INFO:tasks.cephadm:Writing seed config... 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T13:39:37.036 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T13:39:37.037 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T13:39:37.038 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T13:39:37.039 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T13:39:37.039 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T13:39:37.039 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T13:39:37.039 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T13:39:37.039 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T13:39:37.039 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:39:37.040 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T13:39:37.065 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 67c4c1b6-cff5-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T13:39:37.065 DEBUG:teuthology.orchestra.run.smithi158:mon.smithi158> sudo journalctl -f -n 0 -u ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service 2024-02-20T13:39:37.107 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T13:39:37.107 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.158 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:39:37.137 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:-- Logs begin at Tue 2024-02-20 13:28:29 UTC. -- 2024-02-20T13:39:37.317 INFO:teuthology.orchestra.run.smithi158.stderr:-------------------------------------------------------------------------------- 2024-02-20T13:39:37.318 INFO:teuthology.orchestra.run.smithi158.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '67c4c1b6-cff5-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.158', '--skip-admin-label'] 2024-02-20T13:39:37.338 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:39:37.351 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying podman|docker is present... 2024-02-20T13:39:37.371 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:39:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying lvm2 is present... 2024-02-20T13:39:37.385 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying time synchronization is in place... 2024-02-20T13:39:37.391 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T13:39:37.397 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:39:37.403 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: enabled 2024-02-20T13:39:37.408 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: active 2024-02-20T13:39:37.409 INFO:teuthology.orchestra.run.smithi158.stderr:Unit chronyd.service is enabled and running 2024-02-20T13:39:37.409 INFO:teuthology.orchestra.run.smithi158.stderr:Repeating the final host check... 2024-02-20T13:39:37.429 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:39:37.441 INFO:teuthology.orchestra.run.smithi158.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-20T13:39:37.441 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl is present 2024-02-20T13:39:37.441 INFO:teuthology.orchestra.run.smithi158.stderr:lvcreate is present 2024-02-20T13:39:37.446 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T13:39:37.451 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:39:37.457 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: enabled 2024-02-20T13:39:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: active 2024-02-20T13:39:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:Unit chronyd.service is enabled and running 2024-02-20T13:39:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:Host looks OK 2024-02-20T13:39:37.463 INFO:teuthology.orchestra.run.smithi158.stderr:Cluster fsid: 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:39:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:Acquiring lock 140433779116520 on /run/cephadm/67c4c1b6-cff5-11ee-95bb-87774f69a715.lock 2024-02-20T13:39:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:Lock 140433779116520 acquired on /run/cephadm/67c4c1b6-cff5-11ee-95bb-87774f69a715.lock 2024-02-20T13:39:37.464 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying IP 172.21.15.158 port 3300 ... 2024-02-20T13:39:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying IP 172.21.15.158 port 6789 ... 2024-02-20T13:39:37.465 INFO:teuthology.orchestra.run.smithi158.stderr:Base mon IP(s) is [172.21.15.158:3300, 172.21.15.158:6789], mon addrv is [v2:172.21.15.158:3300,v1:172.21.15.158:6789] 2024-02-20T13:39:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.158 metric 100 2024-02-20T13:39:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.158 metric 100 2024-02-20T13:39:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T13:39:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-20T13:39:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:726f/64 scope link 2024-02-20T13:39:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T13:39:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:Mon IP `172.21.15.158` is in CIDR network `172.21.0.0/20` 2024-02-20T13:39:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:Mon IP `172.21.15.158` is in CIDR network `172.21.0.0/20` 2024-02-20T13:39:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-20T13:39:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T13:39:37.476 INFO:teuthology.orchestra.run.smithi158.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:39:37.552 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:39:43.949 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Getting image source signatures 2024-02-20T13:39:43.949 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-20T13:39:43.949 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2024-02-20T13:39:43.949 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-02-20T13:39:44.605 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: Writing manifest to image destination 2024-02-20T13:39:45.645 INFO:teuthology.orchestra.run.smithi158.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-02-20T13:39:47.165 INFO:teuthology.orchestra.run.smithi158.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-02-20T13:39:47.758 INFO:teuthology.orchestra.run.smithi158.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-02-20T13:39:47.758 INFO:teuthology.orchestra.run.smithi158.stderr:Extracting ceph user uid/gid from container image... 2024-02-20T13:39:48.322 INFO:teuthology.orchestra.run.smithi158.stderr:stat: 167 167 2024-02-20T13:39:48.850 INFO:teuthology.orchestra.run.smithi158.stderr:Creating initial keys... 2024-02-20T13:39:49.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-authtool: AQClq9Rl5YNjExAAVb96Ar1E9HtiykyvHIo99Q== 2024-02-20T13:39:50.586 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-authtool: AQCmq9RloTwsHRAA2ttSF8wpWaDnWtj4PwkGug== 2024-02-20T13:39:51.610 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-authtool: AQCnq9RlgM8WHxAAl22P5Q9apvmH4e2gmCRH3Q== 2024-02-20T13:39:54.904 INFO:teuthology.orchestra.run.smithi158.stderr:Creating initial monmap... 2024-02-20T13:39:58.718 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T13:39:58.718 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:39:58.718 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T13:40:01.635 INFO:teuthology.orchestra.run.smithi158.stderr:monmaptool for smithi158 [v2:172.21.15.158:3300,v1:172.21.15.158:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T13:40:01.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/monmaptool: set fsid to 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:01.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T13:40:01.635 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:40:01.635 INFO:teuthology.orchestra.run.smithi158.stderr:Creating mon... 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.072+0000 7fb47e5c8700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 1 imported monmap: 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: last_changed 2024-02-20T13:39:58.135763+0000 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: created 2024-02-20T13:39:58.135763+0000 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon.smithi158 2024-02-20T13:40:02.188 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 0 /usr/bin/ceph-mon: set fsid to 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Compile date Jul 8 2021 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: DB SUMMARY 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi158/store.db dir, Total Num: 0, files: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi158/store.db: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.create_if_missing: 1 2024-02-20T13:40:02.189 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.env: 0x56085f6931c0 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.fs: Posix File System 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.info_log: 0x560860f0cc20 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.statistics: (nil) 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.use_fsync: 0 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.073+0000 7fb47e5c8700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T13:40:02.190 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.db_log_dir: 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi158/store.db 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T13:40:02.191 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.write_buffer_manager: 0x560860f15530 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T13:40:02.192 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.unordered_write: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.row_cache: None 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.wal_filter: None 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T13:40:02.193 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T13:40:02.194 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_open_files: -1 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T13:40:02.195 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Compression algorithms supported: 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kZSTD supported: 0 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.196 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi158/store.db/MANIFEST-000001 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.merge_operator: 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_filter: None 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560860e26160) 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T13:40:02.197 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_cache: 0x560860e5cd10 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-20T13:40:02.198 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-20T13:40:02.199 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression: NoCompression 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.num_levels: 7 2024-02-20T13:40:02.200 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:40:02.201 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T13:40:02.202 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T13:40:02.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T13:40:02.204 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.ttl: 2592000 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.074+0000 7fb47e5c8700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.075+0000 7fb47e5c8700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi158/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.075+0000 7fb47e5c8700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.076+0000 7fb47e5c8700 4 rocksdb: DB pointer 0x560860f21800 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.076+0000 7fb467451700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.076+0000 7fb467451700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:02.205 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:02.206 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:40:02.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:02.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.077+0000 7fb47e5c8700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.077+0000 7fb47e5c8700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-20T13:40:02.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:40:02.077+0000 7fb47e5c8700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi158 for mon.smithi158 2024-02-20T13:40:02.703 INFO:teuthology.orchestra.run.smithi158.stderr:create mon.smithi158 on 2024-02-20T13:40:02.718 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-20T13:40:02.812 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T13:40:02.905 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target. 2024-02-20T13:40:02.905 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target. 2024-02-20T13:40:03.100 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Failed to reset failed state of unit ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service: Unit ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service not loaded. 2024-02-20T13:40:03.107 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Created symlink /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target.wants/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service → /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@.service. 2024-02-20T13:40:03.735 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: disabled 2024-02-20T13:40:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:40:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:firewalld.service is not enabled 2024-02-20T13:40:03.742 INFO:teuthology.orchestra.run.smithi158.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T13:40:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mon to start... 2024-02-20T13:40:03.743 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mon... 2024-02-20T13:40:04.009 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.714610+0000 mon.smithi158 (mon.0) 1 : cluster [INF] mon.smithi158 is new leader, mons smithi158 in quorum (ranks 0) 2024-02-20T13:40:04.009 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:03 smithi158 systemd[1]: Started Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715. 2024-02-20T13:40:04.315 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.321+0000 7f0d9e4b6700 1 Processor -- start 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.321+0000 7f0d9e4b6700 1 -- start start 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.321+0000 7f0d9e4b6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.321+0000 7f0d9e4b6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d980fb690 con 0x7f0d980fad80 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d9d4b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d9d4b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41588/0 (socket says 172.21.15.158:41588) 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d9d4b4700 1 -- 172.21.15.158:0/4274653487 learned_addr learned my addr 172.21.15.158:0/4274653487 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d9d4b4700 1 -- 172.21.15.158:0/4274653487 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d980fb7d0 con 0x7f0d980fad80 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d9d4b4700 1 --2- 172.21.15.158:0/4274653487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0d9400b5f0 tx=0x7f0d94006ee0).ready entity=mon.0 client_cookie=330599009cea3347 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d87fff700 1 -- 172.21.15.158:0/4274653487 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9400d600 con 0x7f0d980fad80 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d87fff700 1 -- 172.21.15.158:0/4274653487 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9400d760 con 0x7f0d980fad80 2024-02-20T13:40:04.316 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.322+0000 7f0d87fff700 1 -- 172.21.15.158:0/4274653487 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9400da30 con 0x7f0d980fad80 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/4274653487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 msgr2=0x7f0d980fb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 --2- 172.21.15.158:0/4274653487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0d9400b5f0 tx=0x7f0d94006ee0).stop 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/4274653487 shutdown_connections 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 --2- 172.21.15.158:0/4274653487 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d980fb1a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/4274653487 >> 172.21.15.158:0/4274653487 conn(0x7f0d980f64e0 msgr2=0x7f0d980f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/4274653487 shutdown_connections 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.323+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/4274653487 wait complete. 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9e4b6700 1 Processor -- start 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9e4b6700 1 -- start start 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9e4b6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9e4b6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9806adf0 con 0x7f0d980fad80 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9d4b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:04.317 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9d4b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41600/0 (socket says 172.21.15.158:41600) 2024-02-20T13:40:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9d4b4700 1 -- 172.21.15.158:0/2110283759 learned_addr learned my addr 172.21.15.158:0/2110283759 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.324+0000 7f0d9d4b4700 1 -- 172.21.15.158:0/2110283759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9400b200 con 0x7f0d980fad80 2024-02-20T13:40:04.318 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d9d4b4700 1 --2- 172.21.15.158:0/2110283759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0d94006c70 tx=0x7f0d94016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:04.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d94011b40 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d94011ca0 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9401d430 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9806aff0 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9810d770 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f0d9400f070 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.325+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0d9401dc10 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.326+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d98045bb0 con 0x7f0d980fad80 2024-02-20T13:40:04.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.328+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0d94016070 con 0x7f0d980fad80 2024-02-20T13:40:04.390 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.394+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0d980fbfe0 con 0x7f0d980fad80 2024-02-20T13:40:04.390 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.394+0000 7f0d85ffb700 1 -- 172.21.15.158:0/2110283759 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0d9401d6d0 con 0x7f0d980fad80 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: cluster: 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: id: 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: services: 2024-02-20T13:40:04.391 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi158 (age 0.666061s) 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: data: 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-20T13:40:04.392 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-20T13:40:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-20T13:40:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: pgs: 2024-02-20T13:40:04.393 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 msgr2=0x7f0d9810d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 --2- 172.21.15.158:0/2110283759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0d94006c70 tx=0x7f0d94016000).stop 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 shutdown_connections 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 --2- 172.21.15.158:0/2110283759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d980fad80 0x7f0d9810d0c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 >> 172.21.15.158:0/2110283759 conn(0x7f0d980f64e0 msgr2=0x7f0d980fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:04.394 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 shutdown_connections 2024-02-20T13:40:04.395 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:04.395+0000 7f0d9e4b6700 1 -- 172.21.15.158:0/2110283759 wait complete. 2024-02-20T13:40:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:mon is available 2024-02-20T13:40:05.024 INFO:teuthology.orchestra.run.smithi158.stderr:Assimilating anything we can from ceph.conf... 2024-02-20T13:40:05.026 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.729373+0000 mon.smithi158 (mon.0) 2 : cluster [INF] mon.smithi158 is new leader, mons smithi158 in quorum (ranks 0) 2024-02-20T13:40:05.026 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.729511+0000 mon.smithi158 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi158=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0]} 2024-02-20T13:40:05.026 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.730890+0000 mon.smithi158 (mon.0) 4 : cluster [DBG] fsmap 2024-02-20T13:40:05.026 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.732557+0000 mon.smithi158 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T13:40:05.026 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: cluster 2024-02-20T13:40:03.732901+0000 mon.smithi158 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T13:40:05.027 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: audit 2024-02-20T13:40:04.395398+0000 mon.smithi158 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.158:0/2110283759' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.567+0000 7f26d0fa4700 1 Processor -- start 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26d0fa4700 1 -- start start 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26d0fa4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26d0fa4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0ff0e0 con 0x7f26cc027670 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41610/0 (socket says 172.21.15.158:41610) 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26ca59c700 1 -- 172.21.15.158:0/628003759 learned_addr learned my addr 172.21.15.158:0/628003759 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.568+0000 7f26ca59c700 1 -- 172.21.15.158:0/628003759 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc0ff220 con 0x7f26cc027670 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26ca59c700 1 --2- 172.21.15.158:0/628003759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f26c000b5f0 tx=0x7f26c0006ee0).ready entity=mon.0 client_cookie=3a111dfb8b4b1c0f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26c959a700 1 -- 172.21.15.158:0/628003759 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c000d600 con 0x7f26cc027670 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26c959a700 1 -- 172.21.15.158:0/628003759 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c000d760 con 0x7f26cc027670 2024-02-20T13:40:05.566 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26c959a700 1 -- 172.21.15.158:0/628003759 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c000da30 con 0x7f26cc027670 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 -- 172.21.15.158:0/628003759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 msgr2=0x7f26cc0febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 --2- 172.21.15.158:0/628003759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f26c000b5f0 tx=0x7f26c0006ee0).stop 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 -- 172.21.15.158:0/628003759 shutdown_connections 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 --2- 172.21.15.158:0/628003759 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc0febf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 -- 172.21.15.158:0/628003759 >> 172.21.15.158:0/628003759 conn(0x7f26cc0f6690 msgr2=0x7f26cc0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 -- 172.21.15.158:0/628003759 shutdown_connections 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.569+0000 7f26d0fa4700 1 -- 172.21.15.158:0/628003759 wait complete. 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26d0fa4700 1 Processor -- start 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26d0fa4700 1 -- start start 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26d0fa4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26d0fa4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc10aa40 con 0x7f26cc027670 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41618/0 (socket says 172.21.15.158:41618) 2024-02-20T13:40:05.567 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26ca59c700 1 -- 172.21.15.158:0/268983216 learned_addr learned my addr 172.21.15.158:0/268983216 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.570+0000 7f26ca59c700 1 -- 172.21.15.158:0/268983216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c000b200 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26ca59c700 1 --2- 172.21.15.158:0/268983216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f26c0006c70 tx=0x7f26c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c0011b40 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c0011ca0 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26cc10ac40 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c001d430 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc10b0d0 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f26c000f070 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.571+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f26c001dbf0 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.572+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc045bb0 con 0x7f26cc027670 2024-02-20T13:40:05.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.574+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f26c0016070 con 0x7f26cc027670 2024-02-20T13:40:05.629 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.638+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f26cc10c5e0 con 0x7f26cc027670 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.641+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f26c001d5e0 con 0x7f26cc027670 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.641+0000 7f26baffd700 1 -- 172.21.15.158:0/268983216 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+456 (secure 0 0 0) 0x7f26c0036020 con 0x7f26cc027670 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [global] 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: fsid = 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.158:3300,v1:172.21.15.158:6789] 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T13:40:05.635 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [mgr] 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [osd] 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T13:40:05.636 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 msgr2=0x7f26cc10a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 --2- 172.21.15.158:0/268983216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f26c0006c70 tx=0x7f26c0016000).stop 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 shutdown_connections 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 --2- 172.21.15.158:0/268983216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26cc027670 0x7f26cc10a550 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 >> 172.21.15.158:0/268983216 conn(0x7f26cc0f6690 msgr2=0x7f26cc0f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 shutdown_connections 2024-02-20T13:40:05.637 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:05.642+0000 7f26d0fa4700 1 -- 172.21.15.158:0/268983216 wait complete. 2024-02-20T13:40:05.960 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: audit 2024-02-20T13:40:05.639588+0000 mon.smithi158 (mon.0) 8 : audit [INF] from='client.? 172.21.15.158:0/268983216' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T13:40:05.961 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: audit 2024-02-20T13:40:05.642583+0000 mon.smithi158 (mon.0) 9 : audit [INF] from='client.? 172.21.15.158:0/268983216' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T13:40:06.280 INFO:teuthology.orchestra.run.smithi158.stderr:Generating new minimal ceph.conf... 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce759e700 1 Processor -- start 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce759e700 1 -- start start 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce759e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8072030 con 0x7f9ce8071c10 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41622/0 (socket says 172.21.15.158:41622) 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.804+0000 7f9ce659c700 1 -- 172.21.15.158:0/3418329771 learned_addr learned my addr 172.21.15.158:0/3418329771 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce659c700 1 -- 172.21.15.158:0/3418329771 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8070600 con 0x7f9ce8071c10 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce659c700 1 --2- 172.21.15.158:0/3418329771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9cd800b5f0 tx=0x7f9cd8006ee0).ready entity=mon.0 client_cookie=f88cff25ff560412 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce559a700 1 -- 172.21.15.158:0/3418329771 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd800d600 con 0x7f9ce8071c10 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce559a700 1 -- 172.21.15.158:0/3418329771 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9cd800dbc0 con 0x7f9ce8071c10 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce559a700 1 -- 172.21.15.158:0/3418329771 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd8011b90 con 0x7f9ce8071c10 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce759e700 1 -- 172.21.15.158:0/3418329771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 msgr2=0x7f9ce8070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.805+0000 7f9ce759e700 1 --2- 172.21.15.158:0/3418329771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9cd800b5f0 tx=0x7f9cd8006ee0).stop 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- 172.21.15.158:0/3418329771 shutdown_connections 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 --2- 172.21.15.158:0/3418329771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce8070110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- 172.21.15.158:0/3418329771 >> 172.21.15.158:0/3418329771 conn(0x7f9ce80f7610 msgr2=0x7f9ce80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- 172.21.15.158:0/3418329771 shutdown_connections 2024-02-20T13:40:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- 172.21.15.158:0/3418329771 wait complete. 2024-02-20T13:40:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 Processor -- start 2024-02-20T13:40:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- start start 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce759e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce810d2c0 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:41638/0 (socket says 172.21.15.158:41638) 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.806+0000 7f9ce659c700 1 -- 172.21.15.158:0/4144798882 learned_addr learned my addr 172.21.15.158:0/4144798882 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9ce659c700 1 -- 172.21.15.158:0/4144798882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd800b200 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9ce659c700 1 --2- 172.21.15.158:0/4144798882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cd800e040 tx=0x7f9cd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd801b570 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9cd801bb30 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce810d4c0 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd801a820 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce810e100 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f9cd802b070 con 0x7f9ce8071c10 2024-02-20T13:40:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.807+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9cd8024bf0 con 0x7f9ce8071c10 2024-02-20T13:40:06.802 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.808+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce810d950 con 0x7f9ce8071c10 2024-02-20T13:40:06.802 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.810+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9cd801f020 con 0x7f9ce8071c10 2024-02-20T13:40:06.865 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.874+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9ce8102060 con 0x7f9ce8071c10 2024-02-20T13:40:06.865 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.874+0000 7f9cd6ffd700 1 -- 172.21.15.158:0/4144798882 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f9cd8016070 con 0x7f9ce8071c10 2024-02-20T13:40:06.866 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 msgr2=0x7f9ce810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:06.866 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 --2- 172.21.15.158:0/4144798882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cd800e040 tx=0x7f9cd8006ee0).stop 2024-02-20T13:40:06.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 shutdown_connections 2024-02-20T13:40:06.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 --2- 172.21.15.158:0/4144798882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ce8071c10 0x7f9ce810cdd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:06.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 >> 172.21.15.158:0/4144798882 conn(0x7f9ce80f7610 msgr2=0x7f9ce80f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:06.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 shutdown_connections 2024-02-20T13:40:06.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:06.875+0000 7f9ce759e700 1 -- 172.21.15.158:0/4144798882 wait complete. 2024-02-20T13:40:07.184 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:06 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: audit 2024-02-20T13:40:06.875413+0000 mon.smithi158 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.158:0/4144798882' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:40:07.430 INFO:teuthology.orchestra.run.smithi158.stderr:Restarting the monitor... 2024-02-20T13:40:07.694 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:07 smithi158 systemd[1]: Stopping Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715... 2024-02-20T13:40:07.695 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:07 smithi158 bash[38132]: Error: no container with name or ID "ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon.smithi158" found: no such container 2024-02-20T13:40:07.695 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: debug 2024-02-20T13:40:07.703+0000 7ff026d6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi158 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T13:40:08.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[37806]: debug 2024-02-20T13:40:07.703+0000 7ff026d6b700 -1 mon.smithi158@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T13:40:08.448 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:08 smithi158 bash[38171]: ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158 2024-02-20T13:40:08.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:08 smithi158 systemd[1]: ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service: Succeeded. 2024-02-20T13:40:08.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:08 smithi158 systemd[1]: Stopped Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715. 2024-02-20T13:40:08.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:08 smithi158 systemd[1]: Starting Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715... 2024-02-20T13:40:09.113 INFO:teuthology.orchestra.run.smithi158.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-20T13:40:09.345 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.079+0000 7f37161b9700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T13:40:09.345 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.079+0000 7f37161b9700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2024-02-20T13:40:09.345 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.079+0000 7f37161b9700 0 pidfile_write: ignore empty --pid-file 2024-02-20T13:40:09.345 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 0 load: jerasure load: lrc load: isa 2024-02-20T13:40:09.345 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Compile date Jul 8 2021 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: DB SUMMARY 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: CURRENT file: CURRENT 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi158/store.db dir, Total Num: 1, files: 000004.sst 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi158/store.db: 000006.log size: 99996 ; 2024-02-20T13:40:09.346 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.create_if_missing: 0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.env: 0x55f014cca1c0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.fs: Posix File System 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.info_log: 0x55f016b0da40 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.statistics: (nil) 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.use_fsync: 0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T13:40:09.347 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.db_log_dir: 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi158/store.db 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T13:40:09.348 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.write_buffer_manager: 0x55f016b15fb0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T13:40:09.349 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.unordered_write: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.row_cache: None 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.wal_filter: None 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T13:40:09.350 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T13:40:09.351 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.max_open_files: -1 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Compression algorithms supported: 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kZSTD supported: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T13:40:09.352 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.084+0000 7f37161b9700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi158/store.db/MANIFEST-000005 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.merge_operator: 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_filter: None 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f016a26170) 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cache_index_and_filter_blocks: 1 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T13:40:09.353 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: pin_top_level_index_and_filter: 1 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: index_type: 0 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: data_block_index_type: 0 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: index_shortening: 1 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: data_block_hash_table_util_ratio: 0.750000 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: hash_index_allow_collision: 1 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: checksum: 1 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: no_block_cache: 0 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_cache: 0x55f016a5cf10 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_cache_name: BinnedLRUCache 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_cache_options: 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: capacity : 536870912 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: num_shard_bits : 4 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: strict_capacity_limit : 0 2024-02-20T13:40:09.354 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: high_pri_pool_ratio: 0.000 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_cache_compressed: (nil) 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: persistent_cache: (nil) 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_size: 4096 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_size_deviation: 10 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_restart_interval: 16 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: index_block_restart_interval: 1 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: metadata_block_size: 4096 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: partition_filters: 0 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: use_delta_encoding: 1 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: whole_key_filtering: 1 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: verify_compression: 0 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: read_amp_bytes_per_bit: 0 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: format_version: 2 2024-02-20T13:40:09.355 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: enable_index_compression: 1 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: block_align: 0 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression: NoCompression 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.num_levels: 7 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T13:40:09.356 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T13:40:09.357 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T13:40:09.358 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T13:40:09.358 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T13:40:09.359 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T13:40:09.360 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T13:40:09.361 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.ttl: 2592000 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi158/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708436409086697, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.085+0000 7f37161b9700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.086+0000 7f37161b9700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.087+0000 7f37161b9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708436409087968, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 95375, "table_properties": {"data_size": 93470, "index_size": 382, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12918, "raw_average_key_size": 49, "raw_value_size": 86364, "raw_average_value_size": 332, "num_data_blocks": 12, "num_entries": 260, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708436409, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.087+0000 7f37161b9700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.087+0000 7f37161b9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708436409088479, "job": 1, "event": "recovery_finished"} 2024-02-20T13:40:09.362 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.088+0000 7f37161b9700 4 rocksdb: DB pointer 0x55f016b21800 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.088+0000 7f36fcfc1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.088+0000 7f36fcfc1700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** DB Stats ** 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** Compaction Stats [default] ** 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:09.363 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: L0 2/0 94.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Sum 2/0 94.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** Compaction Stats [default] ** 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:40:09.364 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Cumulative compaction: 0.00 GB write, 25.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Interval compaction: 0.00 GB write, 25.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** Compaction Stats [default] ** 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: L0 2/0 94.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Sum 2/0 94.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** Compaction Stats [default] ** 2024-02-20T13:40:09.365 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.9 0.00 0.00 1 0.001 0 0 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:40:09.366 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Cumulative compaction: 0.00 GB write, 24.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:40:09.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.089+0000 7f37161b9700 0 starting mon.smithi158 rank 0 at public addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] at bind addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi158 fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 1 mon.smithi158@-1(???) e1 preinit fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).mds e1 new map 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).mds e1 print_map 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: e1 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: legacy client fscid: -1 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: No filesystems configured 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: 2024-02-20T13:40:09.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 0 mon.smithi158@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:06.875413+0000 mon.smithi158 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.158:0/4144798882' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.091+0000 7f37161b9700 1 mon.smithi158@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expand map: {default=false} 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta from 'false' to 'false' 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expanded map: {default=false} 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expand map: {default=info} 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta from 'info' to 'info' 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expanded map: {default=info} 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expand map: {default=daemon} 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-20T13:40:09.369 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expand map: {default=debug} 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:40:09.093+0000 7f37161b9700 20 expand_channel_meta expanded map: {default=debug} 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 bash[38296]: 9b0da332cea7d290f154dfcaa4c2c6693467c0ba22697f99a3a9252db69daf77 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 systemd[1]: Started Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715. 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:09.108335+0000 mon.smithi158 (mon.0) 1 : cluster [INF] mon.smithi158 is new leader, mons smithi158 in quorum (ranks 0) 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:09.108381+0000 mon.smithi158 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi158=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0]} 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:09.108409+0000 mon.smithi158 (mon.0) 3 : cluster [DBG] fsmap 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:09.108434+0000 mon.smithi158 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T13:40:09.370 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:09.108734+0000 mon.smithi158 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T13:40:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.675+0000 7f5ab69dc700 1 Processor -- start 2024-02-20T13:40:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.675+0000 7f5ab69dc700 1 -- start start 2024-02-20T13:40:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.675+0000 7f5ab69dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:09.671 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.675+0000 7f5ab69dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0072090 con 0x7f5ab0071c70 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56992/0 (socket says 172.21.15.158:56992) 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaffff700 1 -- 172.21.15.158:0/2860984101 learned_addr learned my addr 172.21.15.158:0/2860984101 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaffff700 1 -- 172.21.15.158:0/2860984101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab0070660 con 0x7f5ab0071c70 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaffff700 1 --2- 172.21.15.158:0/2860984101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5aa400b5f0 tx=0x7f5aa4006ee0).ready entity=mon.0 client_cookie=e1e506d77eb7faa4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.676+0000 7f5aaeffd700 1 -- 172.21.15.158:0/2860984101 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa400d600 con 0x7f5ab0071c70 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5aaeffd700 1 -- 172.21.15.158:0/2860984101 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5aa400dbc0 con 0x7f5ab0071c70 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5aaeffd700 1 -- 172.21.15.158:0/2860984101 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa4011b90 con 0x7f5ab0071c70 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 -- 172.21.15.158:0/2860984101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 msgr2=0x7f5ab0070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 --2- 172.21.15.158:0/2860984101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5aa400b5f0 tx=0x7f5aa4006ee0).stop 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 -- 172.21.15.158:0/2860984101 shutdown_connections 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 --2- 172.21.15.158:0/2860984101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab0070170 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:09.672 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 -- 172.21.15.158:0/2860984101 >> 172.21.15.158:0/2860984101 conn(0x7f5ab00f7780 msgr2=0x7f5ab00f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 -- 172.21.15.158:0/2860984101 shutdown_connections 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.677+0000 7f5ab69dc700 1 -- 172.21.15.158:0/2860984101 wait complete. 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5ab69dc700 1 Processor -- start 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5ab69dc700 1 -- start start 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5ab69dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5ab69dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab010d3d0 con 0x7f5ab0071c70 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5aaffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5aaffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56998/0 (socket says 172.21.15.158:56998) 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.678+0000 7f5aaffff700 1 -- 172.21.15.158:0/4026358470 learned_addr learned my addr 172.21.15.158:0/4026358470 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aaffff700 1 -- 172.21.15.158:0/4026358470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa400b200 con 0x7f5ab0071c70 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aaffff700 1 --2- 172.21.15.158:0/4026358470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5aa400e040 tx=0x7f5aa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa401b570 con 0x7f5ab0071c70 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5aa401bb30 con 0x7f5ab0071c70 2024-02-20T13:40:09.673 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa401a820 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab010d5d0 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab010da60 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f5aa402b070 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.679+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5aa4024bf0 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.680+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab0045bb0 con 0x7f5ab0071c70 2024-02-20T13:40:09.674 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.682+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5aa401f020 con 0x7f5ab0071c70 2024-02-20T13:40:09.737 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.746+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f5ab0102170 con 0x7f5ab0071c70 2024-02-20T13:40:09.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.748+0000 7f5aacff9700 1 -- 172.21.15.158:0/4026358470 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5aa4016070 con 0x7f5ab0071c70 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.748+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 msgr2=0x7f5ab010cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.748+0000 7f5ab69dc700 1 --2- 172.21.15.158:0/4026358470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5aa400e040 tx=0x7f5aa4006ee0).stop 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.749+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 shutdown_connections 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.749+0000 7f5ab69dc700 1 --2- 172.21.15.158:0/4026358470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab0071c70 0x7f5ab010cee0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.749+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 >> 172.21.15.158:0/4026358470 conn(0x7f5ab00f7780 msgr2=0x7f5ab00f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.749+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 shutdown_connections 2024-02-20T13:40:09.741 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:09.749+0000 7f5ab69dc700 1 -- 172.21.15.158:0/4026358470 wait complete. 2024-02-20T13:40:10.311 INFO:teuthology.orchestra.run.smithi158.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-20T13:40:10.311 INFO:teuthology.orchestra.run.smithi158.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:40:10.311 INFO:teuthology.orchestra.run.smithi158.stderr:Creating mgr... 2024-02-20T13:40:10.312 INFO:teuthology.orchestra.run.smithi158.stderr:Verifying port 9283 ... 2024-02-20T13:40:10.436 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Failed to reset failed state of unit ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mgr.smithi158.gmoobf.service: Unit ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mgr.smithi158.gmoobf.service not loaded. 2024-02-20T13:40:10.442 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: Created symlink /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715.target.wants/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mgr.smithi158.gmoobf.service → /etc/systemd/system/ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@.service. 2024-02-20T13:40:11.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:09.750001+0000 mon.smithi158 (mon.0) 6 : audit [INF] from='client.? 172.21.15.158:0/4026358470' entity='client.admin' 2024-02-20T13:40:11.144 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: disabled 2024-02-20T13:40:11.153 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:40:11.154 INFO:teuthology.orchestra.run.smithi158.stderr:firewalld.service is not enabled 2024-02-20T13:40:11.154 INFO:teuthology.orchestra.run.smithi158.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T13:40:11.163 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: disabled 2024-02-20T13:40:11.171 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:40:11.172 INFO:teuthology.orchestra.run.smithi158.stderr:firewalld.service is not enabled 2024-02-20T13:40:11.172 INFO:teuthology.orchestra.run.smithi158.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T13:40:11.172 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mgr to start... 2024-02-20T13:40:11.172 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mgr... 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.710+0000 7f72466c3700 1 Processor -- start 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.710+0000 7f72466c3700 1 -- start start 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f72466c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f72466c3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400fca60 con 0x7f72400275f0 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f723ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f723ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57022/0 (socket says 172.21.15.158:57022) 2024-02-20T13:40:11.705 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f723ffff700 1 -- 172.21.15.158:0/588718800 learned_addr learned my addr 172.21.15.158:0/588718800 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f723ffff700 1 -- 172.21.15.158:0/588718800 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240101560 con 0x7f72400275f0 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.711+0000 7f723ffff700 1 --2- 172.21.15.158:0/588718800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f723000b5f0 tx=0x7f7230006ee0).ready entity=mon.0 client_cookie=a072fa5bb75be2b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f723effd700 1 -- 172.21.15.158:0/588718800 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f723000d600 con 0x7f72400275f0 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f723effd700 1 -- 172.21.15.158:0/588718800 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f723000dbc0 con 0x7f72400275f0 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f723effd700 1 -- 172.21.15.158:0/588718800 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7230011b90 con 0x7f72400275f0 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 -- 172.21.15.158:0/588718800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 msgr2=0x7f7240101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 --2- 172.21.15.158:0/588718800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f723000b5f0 tx=0x7f7230006ee0).stop 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 -- 172.21.15.158:0/588718800 shutdown_connections 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 --2- 172.21.15.158:0/588718800 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f7240101070 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 -- 172.21.15.158:0/588718800 >> 172.21.15.158:0/588718800 conn(0x7f72400f6690 msgr2=0x7f72400f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 -- 172.21.15.158:0/588718800 shutdown_connections 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.712+0000 7f72466c3700 1 -- 172.21.15.158:0/588718800 wait complete. 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f72466c3700 1 Processor -- start 2024-02-20T13:40:11.706 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f72466c3700 1 -- start start 2024-02-20T13:40:11.707 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f72466c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:11.707 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f72466c3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010d400 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f723ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f723ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57026/0 (socket says 172.21.15.158:57026) 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f723ffff700 1 -- 172.21.15.158:0/1623122760 learned_addr learned my addr 172.21.15.158:0/1623122760 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.713+0000 7f723ffff700 1 -- 172.21.15.158:0/1623122760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723000b200 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723ffff700 1 --2- 172.21.15.158:0/1623122760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7230006800 tx=0x7f7230006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f723000d9c0 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f723001b870 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724010d600 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f723001a740 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724010da90 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f723002b070 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.714+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7230024af0 con 0x7f72400275f0 2024-02-20T13:40:11.708 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.715+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240045bb0 con 0x7f72400275f0 2024-02-20T13:40:11.709 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.717+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f723001f020 con 0x7f72400275f0 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.786+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7240102140 con 0x7f72400275f0 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.786+0000 7f723cff9700 1 -- 172.21.15.158:0/1623122760 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f7230016070 con 0x7f72400275f0 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsid": "67c4c1b6-cff5-11ee-95bb-87774f69a715", 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:40:11.785 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:40:11.786 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:40:11.786 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:40:11.786 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.786 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:40:11.786 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 0 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "smithi158" 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:40:11.788 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "available": false, 2024-02-20T13:40:11.789 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:40:03.730161+0000", 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:11.790 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 msgr2=0x7f724010cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 --2- 172.21.15.158:0/1623122760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7230006800 tx=0x7f7230006ee0).stop 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 shutdown_connections 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 --2- 172.21.15.158:0/1623122760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72400275f0 0x7f724010cf10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 >> 172.21.15.158:0/1623122760 conn(0x7f72400f6690 msgr2=0x7f72400f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 shutdown_connections 2024-02-20T13:40:11.791 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:11.788+0000 7f72466c3700 1 -- 172.21.15.158:0/1623122760 wait complete. 2024-02-20T13:40:12.253 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:11.788503+0000 mon.smithi158 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.158:0/1623122760' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:40:12.531 INFO:teuthology.orchestra.run.smithi158.stderr:mgr not available, waiting (1/15)... 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.093+0000 7fa4caf78700 1 Processor -- start 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.093+0000 7fa4caf78700 1 -- start start 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.093+0000 7fa4caf78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.093+0000 7fa4caf78700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40fca60 con 0x7fa4c40275f0 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c8d14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c8d14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57034/0 (socket says 172.21.15.158:57034) 2024-02-20T13:40:15.090 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c8d14700 1 -- 172.21.15.158:0/1243882048 learned_addr learned my addr 172.21.15.158:0/1243882048 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c8d14700 1 -- 172.21.15.158:0/1243882048 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c4101560 con 0x7fa4c40275f0 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c8d14700 1 --2- 172.21.15.158:0/1243882048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa4b800b5f0 tx=0x7fa4b8006ee0).ready entity=mon.0 client_cookie=7117ba18551810d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c37fe700 1 -- 172.21.15.158:0/1243882048 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b800d600 con 0x7fa4c40275f0 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.094+0000 7fa4c37fe700 1 -- 172.21.15.158:0/1243882048 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa4b800dbc0 con 0x7fa4c40275f0 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4c37fe700 1 -- 172.21.15.158:0/1243882048 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b8011b90 con 0x7fa4c40275f0 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 -- 172.21.15.158:0/1243882048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 msgr2=0x7fa4c4101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 --2- 172.21.15.158:0/1243882048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa4b800b5f0 tx=0x7fa4b8006ee0).stop 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 -- 172.21.15.158:0/1243882048 shutdown_connections 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 --2- 172.21.15.158:0/1243882048 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c4101070 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 -- 172.21.15.158:0/1243882048 >> 172.21.15.158:0/1243882048 conn(0x7fa4c40f6690 msgr2=0x7fa4c40f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 -- 172.21.15.158:0/1243882048 shutdown_connections 2024-02-20T13:40:15.091 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.095+0000 7fa4caf78700 1 -- 172.21.15.158:0/1243882048 wait complete. 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.096+0000 7fa4caf78700 1 Processor -- start 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.096+0000 7fa4caf78700 1 -- start start 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.096+0000 7fa4caf78700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.096+0000 7fa4caf78700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c410d400 con 0x7fa4c40275f0 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.096+0000 7fa4c8d14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c8d14700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57042/0 (socket says 172.21.15.158:57042) 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c8d14700 1 -- 172.21.15.158:0/1924258028 learned_addr learned my addr 172.21.15.158:0/1924258028 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c8d14700 1 -- 172.21.15.158:0/1924258028 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b800b200 con 0x7fa4c40275f0 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c8d14700 1 --2- 172.21.15.158:0/1924258028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa4b8006800 tx=0x7fa4b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:15.092 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b800d9c0 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c410d600 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa4b801b870 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b801a740 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.097+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c410da90 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.098+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fa4b802b070 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.098+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa4b8024b90 con 0x7fa4c40275f0 2024-02-20T13:40:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.098+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c4045bb0 con 0x7fa4c40275f0 2024-02-20T13:40:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.101+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa4b801f020 con 0x7fa4c40275f0 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.169+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa4c4102140 con 0x7fa4c40275f0 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.169+0000 7fa4c17fa700 1 -- 172.21.15.158:0/1924258028 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7fa4b8016070 con 0x7fa4c40275f0 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsid": "67c4c1b6-cff5-11ee-95bb-87774f69a715", 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:40:15.167 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 0 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "smithi158" 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:15.168 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:40:15.169 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.170 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "available": false, 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:15.171 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:40:03.730161+0000", 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.170+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 msgr2=0x7fa4c410cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.170+0000 7fa4caf78700 1 --2- 172.21.15.158:0/1924258028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa4b8006800 tx=0x7fa4b8006ee0).stop 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.170+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 shutdown_connections 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.170+0000 7fa4caf78700 1 --2- 172.21.15.158:0/1924258028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4c40275f0 0x7fa4c410cf10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:15.172 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.170+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 >> 172.21.15.158:0/1924258028 conn(0x7fa4c40f6690 msgr2=0x7fa4c40f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.171+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 shutdown_connections 2024-02-20T13:40:15.173 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:15.171+0000 7fa4caf78700 1 -- 172.21.15.158:0/1924258028 wait complete. 2024-02-20T13:40:15.491 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:15.171553+0000 mon.smithi158 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.158:0/1924258028' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:40:15.745 INFO:teuthology.orchestra.run.smithi158.stderr:mgr not available, waiting (2/15)... 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:16.143888+0000 mon.smithi158 (mon.0) 9 : cluster [INF] Activating manager daemon smithi158.gmoobf 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:16.146577+0000 mon.smithi158 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi158.gmoobf(active, starting, since 0.00277251s) 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.147566+0000 mon.smithi158 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.147761+0000 mon.smithi158 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.147942+0000 mon.smithi158 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.148111+0000 mon.smithi158 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata", "id": "smithi158"}]: dispatch 2024-02-20T13:40:16.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.148287+0000 mon.smithi158 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mgr metadata", "who": "smithi158.gmoobf", "id": "smithi158.gmoobf"}]: dispatch 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:16.149845+0000 mon.smithi158 (mon.0) 16 : cluster [INF] Manager daemon smithi158.gmoobf is now available 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.156700+0000 mon.smithi158 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.161821+0000 mon.smithi158 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.162956+0000 mon.smithi158 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.167416+0000 mon.smithi158 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:16.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:16.169290+0000 mon.smithi158 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.158:0/2545248405' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:18.287 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:17.150298+0000 mon.smithi158 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi158.gmoobf(active, since 1.00649s) 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde22922700 1 Processor -- start 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde22922700 1 -- start start 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde22922700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde22922700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0fb7f0 con 0x7fde1c0faee0 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57094/0 (socket says 172.21.15.158:57094) 2024-02-20T13:40:18.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde1bfff700 1 -- 172.21.15.158:0/4118569342 learned_addr learned my addr 172.21.15.158:0/4118569342 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:18.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.313+0000 7fde1bfff700 1 -- 172.21.15.158:0/4118569342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0fb930 con 0x7fde1c0faee0 2024-02-20T13:40:18.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde1bfff700 1 --2- 172.21.15.158:0/4118569342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fde0c00b5f0 tx=0x7fde0c006ee0).ready entity=mon.0 client_cookie=1fc8ac186012aad6 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde1affd700 1 -- 172.21.15.158:0/4118569342 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0c00d600 con 0x7fde1c0faee0 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde1affd700 1 -- 172.21.15.158:0/4118569342 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fde0c00dbc0 con 0x7fde1c0faee0 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde1affd700 1 -- 172.21.15.158:0/4118569342 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0c011b90 con 0x7fde1c0faee0 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 -- 172.21.15.158:0/4118569342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 msgr2=0x7fde1c0fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 --2- 172.21.15.158:0/4118569342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fde0c00b5f0 tx=0x7fde0c006ee0).stop 2024-02-20T13:40:18.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 -- 172.21.15.158:0/4118569342 shutdown_connections 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 --2- 172.21.15.158:0/4118569342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c0fb300 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 -- 172.21.15.158:0/4118569342 >> 172.21.15.158:0/4118569342 conn(0x7fde1c0f6690 msgr2=0x7fde1c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 -- 172.21.15.158:0/4118569342 shutdown_connections 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.314+0000 7fde22922700 1 -- 172.21.15.158:0/4118569342 wait complete. 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde22922700 1 Processor -- start 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde22922700 1 -- start start 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde22922700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde22922700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0275f0 con 0x7fde1c0faee0 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde1bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:57102/0 (socket says 172.21.15.158:57102) 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.315+0000 7fde1bfff700 1 -- 172.21.15.158:0/3435229557 learned_addr learned my addr 172.21.15.158:0/3435229557 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde1bfff700 1 -- 172.21.15.158:0/3435229557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c00b200 con 0x7fde1c0faee0 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde1bfff700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fde0c006800 tx=0x7fde0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:18.310 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0c00de50 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c0277f0 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fde0c01b870 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0c01a7a0 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.316+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c10e0c0 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 3) v1 ==== 37004+0+0 (secure 0 0 0) 0x7fde0c00f070 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde18ff9700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 0x7fde08033370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fde0c042a50 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde1b7fe700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 0x7fde08033370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c0591e0 con 0x7fde1c0faee0 2024-02-20T13:40:18.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.317+0000 7fde1b7fe700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 0x7fde08033370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fde100096a0 tx=0x7fde10006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:18.313 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.322+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde0c016030 con 0x7fde1c0faee0 2024-02-20T13:40:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.526+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fde1c0fccf0 con 0x7fde1c0faee0 2024-02-20T13:40:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.527+0000 7fde18ff9700 1 -- 172.21.15.158:0/3435229557 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fde0c032090 con 0x7fde1c0faee0 2024-02-20T13:40:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsid": "67c4c1b6-cff5-11ee-95bb-87774f69a715", 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 0 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "smithi158" 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-20T13:40:18.526 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:40:18.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ], 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:40:03.730161+0000", 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: }, 2024-02-20T13:40:18.529 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.529+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 msgr2=0x7fde08033370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.529+0000 7fde22922700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 0x7fde08033370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fde100096a0 tx=0x7fde10006b40).stop 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.529+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 msgr2=0x7fde1c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.529+0000 7fde22922700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fde0c006800 tx=0x7fde0c006ee0).stop 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 shutdown_connections 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fde08030eb0 0x7fde08033370 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 --2- 172.21.15.158:0/3435229557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde1c0faee0 0x7fde1c10da10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 >> 172.21.15.158:0/3435229557 conn(0x7fde1c0f6690 msgr2=0x7fde1c0f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 shutdown_connections 2024-02-20T13:40:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:18.530+0000 7fde22922700 1 -- 172.21.15.158:0/3435229557 wait complete. 2024-02-20T13:40:19.143 INFO:teuthology.orchestra.run.smithi158.stderr:mgr is available 2024-02-20T13:40:19.418 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:18.528719+0000 mon.smithi158 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.158:0/3435229557' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe8619d4700 1 Processor -- start 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe8619d4700 1 -- start start 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe8619d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe8619d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0fced0 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe85affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe85affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38932/0 (socket says 172.21.15.158:38932) 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe85affd700 1 -- 172.21.15.158:0/1039084346 learned_addr learned my addr 172.21.15.158:0/1039084346 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe85affd700 1 -- 172.21.15.158:0/1039084346 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c0fd010 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.693+0000 7fe85affd700 1 --2- 172.21.15.158:0/1039084346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe84c00b5f0 tx=0x7fe84c006ee0).ready entity=mon.0 client_cookie=3713d6da92aea674 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:19.689 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe859ffb700 1 -- 172.21.15.158:0/1039084346 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe84c00d600 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe859ffb700 1 -- 172.21.15.158:0/1039084346 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe84c00dbc0 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe859ffb700 1 -- 172.21.15.158:0/1039084346 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe84c011b90 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 -- 172.21.15.158:0/1039084346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 msgr2=0x7fe85c0fc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1039084346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe84c00b5f0 tx=0x7fe84c006ee0).stop 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 -- 172.21.15.158:0/1039084346 shutdown_connections 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1039084346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c0fc9e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 -- 172.21.15.158:0/1039084346 >> 172.21.15.158:0/1039084346 conn(0x7fe85c0f5e30 msgr2=0x7fe85c0f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 -- 172.21.15.158:0/1039084346 shutdown_connections 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.694+0000 7fe8619d4700 1 -- 172.21.15.158:0/1039084346 wait complete. 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe8619d4700 1 Processor -- start 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe8619d4700 1 -- start start 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe8619d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe8619d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c10d440 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe85affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:19.690 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe85affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38938/0 (socket says 172.21.15.158:38938) 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe85affd700 1 -- 172.21.15.158:0/1150911094 learned_addr learned my addr 172.21.15.158:0/1150911094 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.695+0000 7fe85affd700 1 -- 172.21.15.158:0/1150911094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c00b200 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe85affd700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe84c000c00 tx=0x7fe84c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe84c011560 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe84c01b870 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c10d640 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe84c01a690 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.696+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c10dad0 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7fe84c02b070 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe8437fe700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 0x7fe848033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe84c0438a0 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe85a7fc700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 0x7fe848033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c045bb0 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.691 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.697+0000 7fe85a7fc700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 0x7fe848033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe8500096a0 tx=0x7fe850006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:19.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.701+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe84c016020 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.844 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.853+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe85c102140 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.854+0000 7fe8437fe700 1 -- 172.21.15.158:0/1150911094 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7fe84c01f070 con 0x7fe85c0fc5e0 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [global] 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: fsid = 67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.848 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [mgr] 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: [osd] 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T13:40:19.849 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 msgr2=0x7fe848033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 0x7fe848033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe8500096a0 tx=0x7fe850006b40).stop 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 msgr2=0x7fe85c10cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe84c000c00 tx=0x7fe84c006ee0).stop 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 shutdown_connections 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fe848030fa0 0x7fe848033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 --2- 172.21.15.158:0/1150911094 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c0fc5e0 0x7fe85c10cf50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:19.850 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 >> 172.21.15.158:0/1150911094 conn(0x7fe85c0f5e30 msgr2=0x7fe85c0f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:19.851 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 shutdown_connections 2024-02-20T13:40:19.851 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:19.855+0000 7fe8619d4700 1 -- 172.21.15.158:0/1150911094 wait complete. 2024-02-20T13:40:20.392 INFO:teuthology.orchestra.run.smithi158.stderr:Enabling cephadm module... 2024-02-20T13:40:20.394 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:19.149479+0000 mon.smithi158 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi158.gmoobf(active, since 3s) 2024-02-20T13:40:20.394 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:19.855087+0000 mon.smithi158 (mon.0) 25 : audit [INF] from='client.? 172.21.15.158:0/1150911094' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f4750623700 1 Processor -- start 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f4750623700 1 -- start start 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f4750623700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f4750623700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47480ff120 con 0x7f474806ae10 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f474e3bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f474e3bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38942/0 (socket says 172.21.15.158:38942) 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f474e3bf700 1 -- 172.21.15.158:0/3325809751 learned_addr learned my addr 172.21.15.158:0/3325809751 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f474e3bf700 1 -- 172.21.15.158:0/3325809751 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47480ff260 con 0x7f474806ae10 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.909+0000 7f474e3bf700 1 --2- 172.21.15.158:0/3325809751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f474400b5f0 tx=0x7f4744006ee0).ready entity=mon.0 client_cookie=3c7cf48fed4522f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f474d3bd700 1 -- 172.21.15.158:0/3325809751 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474400d600 con 0x7f474806ae10 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f474d3bd700 1 -- 172.21.15.158:0/3325809751 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f474400dbc0 con 0x7f474806ae10 2024-02-20T13:40:20.905 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f474d3bd700 1 -- 172.21.15.158:0/3325809751 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4744011b90 con 0x7f474806ae10 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 -- 172.21.15.158:0/3325809751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 msgr2=0x7f47480fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 --2- 172.21.15.158:0/3325809751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f474400b5f0 tx=0x7f4744006ee0).stop 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 -- 172.21.15.158:0/3325809751 shutdown_connections 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 --2- 172.21.15.158:0/3325809751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fec30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 -- 172.21.15.158:0/3325809751 >> 172.21.15.158:0/3325809751 conn(0x7f47480f6690 msgr2=0x7f47480f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 -- 172.21.15.158:0/3325809751 shutdown_connections 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.910+0000 7f4750623700 1 -- 172.21.15.158:0/3325809751 wait complete. 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f4750623700 1 Processor -- start 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f4750623700 1 -- start start 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f4750623700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f4750623700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47480fab30 con 0x7f474806ae10 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f474e3bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f474e3bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38950/0 (socket says 172.21.15.158:38950) 2024-02-20T13:40:20.906 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f474e3bf700 1 -- 172.21.15.158:0/145288974 learned_addr learned my addr 172.21.15.158:0/145288974 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.911+0000 7f474e3bf700 1 -- 172.21.15.158:0/145288974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474400b200 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f474e3bf700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4744006800 tx=0x7f4744006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474400de50 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f474401b870 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47480facd0 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474401a7a0 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.912+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47480fb160 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f474400f070 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748045bb0 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f4736ffd700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 0x7f4730033400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4744042b40 con 0x7f474806ae10 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f474dbbe700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 0x7f4730033400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:20.907 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.913+0000 7f474dbbe700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 0x7f4730033400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:20.908 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:20.917+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4744019af0 con 0x7f474806ae10 2024-02-20T13:40:21.097 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.105+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f47480ffed0 con 0x7f474806ae10 2024-02-20T13:40:21.143 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.151+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f474401b9e0 con 0x7f474806ae10 2024-02-20T13:40:21.144 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.152+0000 7f4736ffd700 1 -- 172.21.15.158:0/145288974 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4744010540 con 0x7f474806ae10 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 msgr2=0x7f4730033400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 0x7f4730033400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).stop 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 msgr2=0x7f47480fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4744006800 tx=0x7f4744006ee0).stop 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 shutdown_connections 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f4730030f40 0x7f4730033400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 --2- 172.21.15.158:0/145288974 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f474806ae10 0x7f47480fc6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.154+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 >> 172.21.15.158:0/145288974 conn(0x7f47480f6690 msgr2=0x7f47480f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.155+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 shutdown_connections 2024-02-20T13:40:21.147 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:21.155+0000 7f4750623700 1 -- 172.21.15.158:0/145288974 wait complete. 2024-02-20T13:40:21.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:21.107090+0000 mon.smithi158 (mon.0) 26 : audit [INF] from='client.? 172.21.15.158:0/145288974' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T13:40:22.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a5d7ce700 1 Processor -- start 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a5d7ce700 1 -- start start 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a5d7ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a5d7ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580ff140 con 0x7f1a58027610 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38988/0 (socket says 172.21.15.158:38988) 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a56ffd700 1 -- 172.21.15.158:0/3599812690 learned_addr learned my addr 172.21.15.158:0/3599812690 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a56ffd700 1 -- 172.21.15.158:0/3599812690 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580ff280 con 0x7f1a58027610 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.370+0000 7f1a56ffd700 1 --2- 172.21.15.158:0/3599812690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1a4800b5f0 tx=0x7f1a48006ee0).ready entity=mon.0 client_cookie=2a9f3d324e58f578 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a55ffb700 1 -- 172.21.15.158:0/3599812690 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a4800d600 con 0x7f1a58027610 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a55ffb700 1 -- 172.21.15.158:0/3599812690 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1a4800dbc0 con 0x7f1a58027610 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a55ffb700 1 -- 172.21.15.158:0/3599812690 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a48011b90 con 0x7f1a58027610 2024-02-20T13:40:22.365 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/3599812690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 msgr2=0x7f1a580fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/3599812690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1a4800b5f0 tx=0x7f1a48006ee0).stop 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/3599812690 shutdown_connections 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/3599812690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fec50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/3599812690 >> 172.21.15.158:0/3599812690 conn(0x7f1a580f6690 msgr2=0x7f1a580f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/3599812690 shutdown_connections 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.371+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/3599812690 wait complete. 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.372+0000 7f1a5d7ce700 1 Processor -- start 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.372+0000 7f1a5d7ce700 1 -- start start 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a5d7ce700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a5d7ce700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580fab50 con 0x7f1a58027610 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a56ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:38990/0 (socket says 172.21.15.158:38990) 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a56ffd700 1 -- 172.21.15.158:0/338359079 learned_addr learned my addr 172.21.15.158:0/338359079 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:22.366 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a56ffd700 1 -- 172.21.15.158:0/338359079 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4800b200 con 0x7f1a58027610 2024-02-20T13:40:22.367 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a56ffd700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1a48000c00 tx=0x7f1a48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a4801b410 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1a4801b9d0 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a4801a620 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580facf0 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.373+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a580fb180 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.374+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f1a4802b070 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.374+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58045bb0 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.374+0000 7f1a3f7fe700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 0x7f1a440334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.374+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a48042b30 con 0x7f1a58027610 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.374+0000 7f1a567fc700 1 -- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 msgr2=0x7f1a440334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:22.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.375+0000 7f1a567fc700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 0x7f1a440334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:40:22.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.379+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1a48024400 con 0x7f1a58027610 2024-02-20T13:40:22.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:21.154369+0000 mon.smithi158 (mon.0) 27 : audit [INF] from='client.? 172.21.15.158:0/145288974' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T13:40:22.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:21.154420+0000 mon.smithi158 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi158.gmoobf(active, since 5s) 2024-02-20T13:40:22.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.576+0000 7f1a567fc700 1 -- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 msgr2=0x7f1a440334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:22.568 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.576+0000 7f1a567fc700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 0x7f1a440334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:40:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.613+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1a58110d00 con 0x7f1a58027610 2024-02-20T13:40:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.613+0000 7f1a3f7fe700 1 -- 172.21.15.158:0/338359079 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1a48016080 con 0x7f1a58027610 2024-02-20T13:40:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:22.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-20T13:40:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:40:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "active_name": "smithi158.gmoobf", 2024-02-20T13:40:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T13:40:22.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 msgr2=0x7f1a440334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 0x7f1a440334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 msgr2=0x7f1a580fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1a48000c00 tx=0x7f1a48006ee0).stop 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 shutdown_connections 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7f1a44030ff0 0x7f1a440334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 --2- 172.21.15.158:0/338359079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a58027610 0x7f1a580fc6d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 >> 172.21.15.158:0/338359079 conn(0x7f1a580f6690 msgr2=0x7f1a580f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 shutdown_connections 2024-02-20T13:40:22.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:22.615+0000 7f1a5d7ce700 1 -- 172.21.15.158:0/338359079 wait complete. 2024-02-20T13:40:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for the mgr to restart... 2024-02-20T13:40:23.256 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mgr epoch 5... 2024-02-20T13:40:23.258 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:22.615086+0000 mon.smithi158 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.158:0/338359079' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T13:40:23.812 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.808+0000 7fa941eaa700 1 Processor -- start 2024-02-20T13:40:23.812 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa941eaa700 1 -- start start 2024-02-20T13:40:23.812 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa941eaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa941eaa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0fb7f0 con 0x7fa93c0faee0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39006/0 (socket says 172.21.15.158:39006) 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93b7fe700 1 -- 172.21.15.158:0/1540319057 learned_addr learned my addr 172.21.15.158:0/1540319057 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93b7fe700 1 -- 172.21.15.158:0/1540319057 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c0fb930 con 0x7fa93c0faee0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93b7fe700 1 --2- 172.21.15.158:0/1540319057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa92c00b5f0 tx=0x7fa92c006ee0).ready entity=mon.0 client_cookie=4eaf42886db66c44 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93a7fc700 1 -- 172.21.15.158:0/1540319057 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92c00d600 con 0x7fa93c0faee0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93a7fc700 1 -- 172.21.15.158:0/1540319057 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa92c00dbc0 con 0x7fa93c0faee0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.809+0000 7fa93a7fc700 1 -- 172.21.15.158:0/1540319057 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92c011b90 con 0x7fa93c0faee0 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 -- 172.21.15.158:0/1540319057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 msgr2=0x7fa93c0fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 --2- 172.21.15.158:0/1540319057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa92c00b5f0 tx=0x7fa92c006ee0).stop 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 -- 172.21.15.158:0/1540319057 shutdown_connections 2024-02-20T13:40:23.813 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 --2- 172.21.15.158:0/1540319057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c0fb300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 -- 172.21.15.158:0/1540319057 >> 172.21.15.158:0/1540319057 conn(0x7fa93c0f6690 msgr2=0x7fa93c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 -- 172.21.15.158:0/1540319057 shutdown_connections 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.810+0000 7fa941eaa700 1 -- 172.21.15.158:0/1540319057 wait complete. 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa941eaa700 1 Processor -- start 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa941eaa700 1 -- start start 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa941eaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa941eaa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c10d3f0 con 0x7fa93c0faee0 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:39010/0 (socket says 172.21.15.158:39010) 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa93b7fe700 1 -- 172.21.15.158:0/2272891324 learned_addr learned my addr 172.21.15.158:0/2272891324 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:23.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa93b7fe700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c00b200 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.811+0000 7fa93b7fe700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa92c006800 tx=0x7fa92c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.812+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92c00de50 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.812+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa92c01b870 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.812+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c10d5f0 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.812+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92c01a7a0 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.812+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c10da80 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa92c00f070 con 0x7fa93c0faee0 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa923fff700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:23.815 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa92c011610 con 0x7fa928030ff0 2024-02-20T13:40:23.816 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa92c042b10 con 0x7fa93c0faee0 2024-02-20T13:40:23.816 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa93affd700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 msgr2=0x7fa9280334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:23.816 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:23.813+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:40:24.006 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:24.014+0000 7fa93affd700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 msgr2=0x7fa9280334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:24.006 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:24.014+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:40:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:24.415+0000 7fa93affd700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 msgr2=0x7fa9280334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:24.407 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:24.415+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T13:40:25.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:25.216+0000 7fa93affd700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 msgr2=0x7fa9280334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/3810558139 2024-02-20T13:40:25.208 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:25.216+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T13:40:26.351 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:26.358+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7fa92c01b9e0 con 0x7fa93c0faee0 2024-02-20T13:40:26.351 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:26.359+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 msgr2=0x7fa9280334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:40:26.351 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:26.359+0000 7fa923fff700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.360+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7fa92c010dd0 con 0x7fa93c0faee0 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.360+0000 7fa923fff700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 0x7fa928036c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.360+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa92c011610 con 0x7fa928034890 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.360+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 0x7fa928036c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.361+0000 7fa93affd700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 0x7fa928036c80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa9300036d0 tx=0x7fa930006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:27.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.362+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa92c011610 con 0x7fa928034890 2024-02-20T13:40:27.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa93c0591e0 con 0x7fa928034890 2024-02-20T13:40:27.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa923fff700 1 -- 172.21.15.158:0/2272891324 <== mgr.14116 v2:172.21.15.158:6800/1783554617 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa93c0591e0 con 0x7fa928034890 2024-02-20T13:40:27.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 msgr2=0x7fa928036c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 0x7fa928036c80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa9300036d0 tx=0x7fa930006d20).stop 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 msgr2=0x7fa93c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa92c006800 tx=0x7fa92c006ee0).stop 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 shutdown_connections 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa928034890 0x7fa928036c80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa93c0faee0 0x7fa93c10cf00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 --2- 172.21.15.158:0/2272891324 >> [v2:172.21.15.158:6800/3810558139,v1:172.21.15.158:6801/3810558139] conn(0x7fa928030ff0 0x7fa9280334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 >> 172.21.15.158:0/2272891324 conn(0x7fa93c0f6690 msgr2=0x7fa93c0271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:27.362 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 shutdown_connections 2024-02-20T13:40:27.363 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:27.369+0000 7fa941eaa700 1 -- 172.21.15.158:0/2272891324 wait complete. 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:26.308574+0000 mon.smithi158 (mon.0) 30 : cluster [INF] Active manager daemon smithi158.gmoobf restarted 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:26.308996+0000 mon.smithi158 (mon.0) 31 : cluster [INF] Activating manager daemon smithi158.gmoobf 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:26.310587+0000 mon.smithi158 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:26.361629+0000 mon.smithi158 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi158.gmoobf(active, starting, since 0.052712s) 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.362552+0000 mon.smithi158 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata", "id": "smithi158"}]: dispatch 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.363164+0000 mon.smithi158 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mgr metadata", "who": "smithi158.gmoobf", "id": "smithi158.gmoobf"}]: dispatch 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.364095+0000 mon.smithi158 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:40:27.787 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.364318+0000 mon.smithi158 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.364532+0000 mon.smithi158 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:26.378040+0000 mon.smithi158 (mon.0) 39 : cluster [INF] Manager daemon smithi158.gmoobf is now available 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.391048+0000 mon.smithi158 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.393752+0000 mon.smithi158 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.396166+0000 mon.smithi158 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.483785+0000 mon.smithi158 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.484447+0000 mon.smithi158 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.486036+0000 mon.smithi158 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.487438+0000 mon.smithi158 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.497669+0000 mon.smithi158 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:40:27.788 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:26.498821+0000 mon.smithi158 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:40:28.603 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:27.363577+0000 mgr.smithi158.gmoobf (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T13:40:28.603 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:27.364198+0000 mon.smithi158 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi158.gmoobf(active, since 1.05527s) 2024-02-20T13:40:28.603 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:27.370316+0000 mgr.smithi158.gmoobf (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T13:40:28.881 INFO:teuthology.orchestra.run.smithi158.stderr:mgr epoch 5 is available 2024-02-20T13:40:28.882 INFO:teuthology.orchestra.run.smithi158.stderr:Setting orchestrator backend to cephadm... 2024-02-20T13:40:29.341 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.347+0000 7f0c0776c700 1 Processor -- start 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.347+0000 7f0c0776c700 1 -- start start 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.347+0000 7f0c0776c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.347+0000 7f0c0776c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000fca60 con 0x7f0c0006add0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.347+0000 7f0c05508700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0c05508700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56568/0 (socket says 172.21.15.158:56568) 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0c05508700 1 -- 172.21.15.158:0/3763166104 learned_addr learned my addr 172.21.15.158:0/3763166104 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0c05508700 1 -- 172.21.15.158:0/3763166104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00101560 con 0x7f0c0006add0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0c05508700 1 --2- 172.21.15.158:0/3763166104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f0bf000b5f0 tx=0x7f0bf0006ee0).ready entity=mon.0 client_cookie=13a9a1d9fcdbacd6 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0beffff700 1 -- 172.21.15.158:0/3763166104 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf000d600 con 0x7f0c0006add0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0beffff700 1 -- 172.21.15.158:0/3763166104 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0bf000dbc0 con 0x7f0c0006add0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.348+0000 7f0beffff700 1 -- 172.21.15.158:0/3763166104 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf0011b90 con 0x7f0c0006add0 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- 172.21.15.158:0/3763166104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 msgr2=0x7f0c00101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:29.342 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 --2- 172.21.15.158:0/3763166104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f0bf000b5f0 tx=0x7f0bf0006ee0).stop 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- 172.21.15.158:0/3763166104 shutdown_connections 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 --2- 172.21.15.158:0/3763166104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c00101070 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- 172.21.15.158:0/3763166104 >> 172.21.15.158:0/3763166104 conn(0x7f0c000f6690 msgr2=0x7f0c000f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- 172.21.15.158:0/3763166104 shutdown_connections 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- 172.21.15.158:0/3763166104 wait complete. 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 Processor -- start 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.349+0000 7f0c0776c700 1 -- start start 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c0776c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c0776c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0010d400 con 0x7f0c0006add0 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c05508700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c05508700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56574/0 (socket says 172.21.15.158:56574) 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c05508700 1 -- 172.21.15.158:0/326993227 learned_addr learned my addr 172.21.15.158:0/326993227 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:29.343 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c05508700 1 -- 172.21.15.158:0/326993227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf000b200 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c05508700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0bf0006800 tx=0x7f0bf0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf000d9e0 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0010d600 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0bf001b870 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf001a570 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.350+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0010da90 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.351+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0bf002b070 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.351+0000 7f0bedffb700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 0x7f0be8033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.351+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00045bb0 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.351+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0bf0042c10 con 0x7f0c0006add0 2024-02-20T13:40:29.345 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.352+0000 7f0c04d07700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 0x7f0be8033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:29.346 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.352+0000 7f0c04d07700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 0x7f0be8033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:29.347 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.357+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0bf0016020 con 0x7f0c0006add0 2024-02-20T13:40:29.510 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.518+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c0010e3b0 con 0x7f0be8030fa0 2024-02-20T13:40:29.515 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.523+0000 7f0bedffb700 1 -- 172.21.15.158:0/326993227 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0c0010e3b0 con 0x7f0be8030fa0 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 msgr2=0x7f0be8033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 0x7f0be8033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).stop 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 msgr2=0x7f0c0010cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0bf0006800 tx=0x7f0bf0006ee0).stop 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 shutdown_connections 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f0be8030fa0 0x7f0be8033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:29.518 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 --2- 172.21.15.158:0/326993227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c0006add0 0x7f0c0010cf10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 >> 172.21.15.158:0/326993227 conn(0x7f0c000f6690 msgr2=0x7f0c000f80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 shutdown_connections 2024-02-20T13:40:29.519 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:29.526+0000 7f0c0776c700 1 -- 172.21.15.158:0/326993227 wait complete. 2024-02-20T13:40:29.786 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:28.366483+0000 mon.smithi158 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi158.gmoobf(active, since 2s) 2024-02-20T13:40:30.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.611+0000 7fba040f5700 1 Processor -- start 2024-02-20T13:40:30.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.611+0000 7fba040f5700 1 -- start start 2024-02-20T13:40:30.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.611+0000 7fba040f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:30.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.611+0000 7fba040f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0ff0e0 con 0x7fb9fc027670 2024-02-20T13:40:30.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba01e91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba01e91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56590/0 (socket says 172.21.15.158:56590) 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba01e91700 1 -- 172.21.15.158:0/3151318590 learned_addr learned my addr 172.21.15.158:0/3151318590 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba01e91700 1 -- 172.21.15.158:0/3151318590 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0ff220 con 0x7fb9fc027670 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba01e91700 1 --2- 172.21.15.158:0/3151318590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb9ec00b5f0 tx=0x7fb9ec006ee0).ready entity=mon.0 client_cookie=24dbc7129e66abf4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba00e8f700 1 -- 172.21.15.158:0/3151318590 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec00d600 con 0x7fb9fc027670 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba00e8f700 1 -- 172.21.15.158:0/3151318590 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb9ec00dbc0 con 0x7fb9fc027670 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.612+0000 7fba00e8f700 1 -- 172.21.15.158:0/3151318590 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec011b90 con 0x7fb9fc027670 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- 172.21.15.158:0/3151318590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 msgr2=0x7fb9fc0febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 --2- 172.21.15.158:0/3151318590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb9ec00b5f0 tx=0x7fb9ec006ee0).stop 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- 172.21.15.158:0/3151318590 shutdown_connections 2024-02-20T13:40:30.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 --2- 172.21.15.158:0/3151318590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0febf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:30.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- 172.21.15.158:0/3151318590 >> 172.21.15.158:0/3151318590 conn(0x7fb9fc0f6690 msgr2=0x7fb9fc0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:30.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- 172.21.15.158:0/3151318590 shutdown_connections 2024-02-20T13:40:30.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- 172.21.15.158:0/3151318590 wait complete. 2024-02-20T13:40:30.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 Processor -- start 2024-02-20T13:40:30.606 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.613+0000 7fba040f5700 1 -- start start 2024-02-20T13:40:30.607 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba040f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:30.607 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba040f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0faaf0 con 0x7fb9fc027670 2024-02-20T13:40:30.607 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba01e91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:30.607 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba01e91700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56606/0 (socket says 172.21.15.158:56606) 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba01e91700 1 -- 172.21.15.158:0/2629570698 learned_addr learned my addr 172.21.15.158:0/2629570698 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba01e91700 1 -- 172.21.15.158:0/2629570698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec00b200 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba01e91700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb9ec006800 tx=0x7fb9ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec00de50 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb9ec01b870 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec01a7a0 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc0fac90 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.614+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc0fb120 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.615+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fb9ec00f070 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.615+0000 7fb9ea7fc700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 0x7fb9e40334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.615+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc045bb0 con 0x7fb9fc027670 2024-02-20T13:40:30.608 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.615+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb9ec042b90 con 0x7fb9fc027670 2024-02-20T13:40:30.609 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.615+0000 7fba01690700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 0x7fb9e40334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:30.609 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.616+0000 7fba01690700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 0x7fb9e40334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb9f80096a0 tx=0x7fb9f8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:30.610 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.619+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb9ec016030 con 0x7fb9fc027670 2024-02-20T13:40:30.763 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.772+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9fc0fb360 con 0x7fb9e4030ff0 2024-02-20T13:40:30.763 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.772+0000 7fb9ea7fc700 1 -- 172.21.15.158:0/2629570698 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb9fc0fb360 con 0x7fb9e4030ff0 2024-02-20T13:40:30.763 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: value unchanged 2024-02-20T13:40:30.765 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 msgr2=0x7fb9e40334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:30.765 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 0x7fb9e40334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb9f80096a0 tx=0x7fb9f8006b40).stop 2024-02-20T13:40:30.765 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 msgr2=0x7fb9fc0fc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb9ec006800 tx=0x7fb9ec006ee0).stop 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 shutdown_connections 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fb9e4030ff0 0x7fb9e40334b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 --2- 172.21.15.158:0/2629570698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9fc027670 0x7fb9fc0fc670 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 >> 172.21.15.158:0/2629570698 conn(0x7fb9fc0f6690 msgr2=0x7fb9fc0f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 shutdown_connections 2024-02-20T13:40:30.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:30.774+0000 7fba040f5700 1 -- 172.21.15.158:0/2629570698 wait complete. 2024-02-20T13:40:30.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:29.521446+0000 mgr.smithi158.gmoobf (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:30.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:29.525367+0000 mon.smithi158 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:30.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:29.529232+0000 mon.smithi158 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:31.370 INFO:teuthology.orchestra.run.smithi158.stderr:Generating ssh key... 2024-02-20T13:40:31.626 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:30.773285+0000 mgr.smithi158.gmoobf (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f650c8dc700 1 Processor -- start 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f650c8dc700 1 -- start start 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f650c8dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f650c8dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6508072030 con 0x7f6508071c10 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f6506d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f6506d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56608/0 (socket says 172.21.15.158:56608) 2024-02-20T13:40:31.888 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f6506d9d700 1 -- 172.21.15.158:0/1951711668 learned_addr learned my addr 172.21.15.158:0/1951711668 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f6506d9d700 1 -- 172.21.15.158:0/1951711668 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6508070600 con 0x7f6508071c10 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.894+0000 7f6506d9d700 1 --2- 172.21.15.158:0/1951711668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f64f800b5f0 tx=0x7f64f8006ee0).ready entity=mon.0 client_cookie=d2e10add1d9fa3ae server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f6505d9b700 1 -- 172.21.15.158:0/1951711668 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f800d600 con 0x7f6508071c10 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f6505d9b700 1 -- 172.21.15.158:0/1951711668 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f64f800dbc0 con 0x7f6508071c10 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f6505d9b700 1 -- 172.21.15.158:0/1951711668 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f8011b90 con 0x7f6508071c10 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 -- 172.21.15.158:0/1951711668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 msgr2=0x7f6508070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 --2- 172.21.15.158:0/1951711668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f64f800b5f0 tx=0x7f64f8006ee0).stop 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 -- 172.21.15.158:0/1951711668 shutdown_connections 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 --2- 172.21.15.158:0/1951711668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f6508070110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 -- 172.21.15.158:0/1951711668 >> 172.21.15.158:0/1951711668 conn(0x7f65080f7610 msgr2=0x7f65080f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 -- 172.21.15.158:0/1951711668 shutdown_connections 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.895+0000 7f650c8dc700 1 -- 172.21.15.158:0/1951711668 wait complete. 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f650c8dc700 1 Processor -- start 2024-02-20T13:40:31.889 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f650c8dc700 1 -- start start 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f650c8dc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f650c8dc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650810d2c0 con 0x7f6508071c10 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f6506d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f6506d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56610/0 (socket says 172.21.15.158:56610) 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f6506d9d700 1 -- 172.21.15.158:0/900412359 learned_addr learned my addr 172.21.15.158:0/900412359 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.896+0000 7f6506d9d700 1 -- 172.21.15.158:0/900412359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f800b200 con 0x7f6508071c10 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f6506d9d700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f64f8000c00 tx=0x7f64f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f8011560 con 0x7f6508071c10 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f64f801b870 con 0x7f6508071c10 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650810d4c0 con 0x7f6508071c10 2024-02-20T13:40:31.890 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f801a700 con 0x7f6508071c10 2024-02-20T13:40:31.891 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.897+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650810e100 con 0x7f6508071c10 2024-02-20T13:40:31.891 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f64f802b070 con 0x7f6508071c10 2024-02-20T13:40:31.891 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650810d950 con 0x7f6508071c10 2024-02-20T13:40:31.892 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f64ef7fe700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 0x7f64f40334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:31.892 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f64f8042c20 con 0x7f6508071c10 2024-02-20T13:40:31.892 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f650659c700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 0x7f64f40334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:31.892 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.898+0000 7f650659c700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 0x7f64f40334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f64fc0096a0 tx=0x7f64fc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:31.892 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:31.902+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f64f802b8d0 con 0x7f6508071c10 2024-02-20T13:40:32.043 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.053+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6508102180 con 0x7f64f4030ff0 2024-02-20T13:40:32.154 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.163+0000 7f64ef7fe700 1 -- 172.21.15.158:0/900412359 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6508102180 con 0x7f64f4030ff0 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 msgr2=0x7f64f40334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 0x7f64f40334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f64fc0096a0 tx=0x7f64fc006b40).stop 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 msgr2=0x7f650810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f64f8000c00 tx=0x7f64f8006ee0).stop 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 shutdown_connections 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64f4030ff0 0x7f64f40334b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 --2- 172.21.15.158:0/900412359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6508071c10 0x7f650810cdd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:32.157 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 >> 172.21.15.158:0/900412359 conn(0x7f65080f7610 msgr2=0x7f65080f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:32.158 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 shutdown_connections 2024-02-20T13:40:32.158 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:32.165+0000 7f650c8dc700 1 -- 172.21.15.158:0/900412359 wait complete. 2024-02-20T13:40:33.367 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.373+0000 7fa872b22700 1 Processor -- start 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.373+0000 7fa872b22700 1 -- start start 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa872b22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa872b22700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c0fca60 con 0x7fa86c0275f0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa8708be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa8708be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56622/0 (socket says 172.21.15.158:56622) 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa8708be700 1 -- 172.21.15.158:0/2081635557 learned_addr learned my addr 172.21.15.158:0/2081635557 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa8708be700 1 -- 172.21.15.158:0/2081635557 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c101560 con 0x7fa86c0275f0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa8708be700 1 --2- 172.21.15.158:0/2081635557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa86000b5f0 tx=0x7fa860006ee0).ready entity=mon.0 client_cookie=68004abf36570b22 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa86b7fe700 1 -- 172.21.15.158:0/2081635557 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86000d600 con 0x7fa86c0275f0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa86b7fe700 1 -- 172.21.15.158:0/2081635557 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa86000dbc0 con 0x7fa86c0275f0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.374+0000 7fa86b7fe700 1 -- 172.21.15.158:0/2081635557 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa860011b90 con 0x7fa86c0275f0 2024-02-20T13:40:33.368 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- 172.21.15.158:0/2081635557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 msgr2=0x7fa86c101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 --2- 172.21.15.158:0/2081635557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa86000b5f0 tx=0x7fa860006ee0).stop 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- 172.21.15.158:0/2081635557 shutdown_connections 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 --2- 172.21.15.158:0/2081635557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c101070 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- 172.21.15.158:0/2081635557 >> 172.21.15.158:0/2081635557 conn(0x7fa86c0f6690 msgr2=0x7fa86c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- 172.21.15.158:0/2081635557 shutdown_connections 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- 172.21.15.158:0/2081635557 wait complete. 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 Processor -- start 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.375+0000 7fa872b22700 1 -- start start 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa872b22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa872b22700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c10d400 con 0x7fa86c0275f0 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8708be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8708be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56636/0 (socket says 172.21.15.158:56636) 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8708be700 1 -- 172.21.15.158:0/346062587 learned_addr learned my addr 172.21.15.158:0/346062587 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8708be700 1 -- 172.21.15.158:0/346062587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86000b200 con 0x7fa86c0275f0 2024-02-20T13:40:33.369 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8708be700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa860006800 tx=0x7fa860006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86000d9c0 con 0x7fa86c0275f0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa86001b870 con 0x7fa86c0275f0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86001a740 con 0x7fa86c0275f0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.376+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c10d600 con 0x7fa86c0275f0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.377+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c10da90 con 0x7fa86c0275f0 2024-02-20T13:40:33.370 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.377+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa86002b070 con 0x7fa86c0275f0 2024-02-20T13:40:33.371 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.377+0000 7fa8697fa700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 0x7fa8500334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:33.371 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.377+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa860042c60 con 0x7fa86c0275f0 2024-02-20T13:40:33.371 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.377+0000 7fa86bfff700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 0x7fa8500334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:33.371 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.378+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c045bb0 con 0x7fa86c0275f0 2024-02-20T13:40:33.371 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.378+0000 7fa86bfff700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 0x7fa8500334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa8580096a0 tx=0x7fa858006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:33.372 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.382+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa86001f070 con 0x7fa86c0275f0 2024-02-20T13:40:33.524 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.533+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa86c0008d0 con 0x7fa850030ff0 2024-02-20T13:40:33.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.534+0000 7fa8697fa700 1 -- 172.21.15.158:0/346062587 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa86c0008d0 con 0x7fa850030ff0 2024-02-20T13:40:33.525 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDawOa5jyiIAx5VzXIF3AJIxD5o4vTDY0KOcaIz7eNkEo7x/Q9GKaR0jUNx6V8rGAYlWEPBRaH5nnkuVJbKj2LwP+jCT3O9OIlk75hIkEdLb/NrKUoTOqtJMWjIbUUyJpEVEJy3eiHKuAWcXOYTG9EO7hm90hJVMzfFch2B4w7V4Uc56RxRJIpAzO9sWxqqHeX8hUIJgspnNPjso3gPVcXTT8v0z/3dOjy5D48urL3NeQY+Bc18BrDGWItUX6OC5d7fmLfhD4BZGIAe5Jym3XgnE9YIqxE3lwFDlTWyYMt9IUYj57fMj59qslGRpYDWBnVRjRYRBcE90kQm2DcHFby0vetRL2t+554Ch1fyGMRG0frLUYck0ZdTNum9YgK8zNLi2ozBuS5fr+HJzDMlBnkxa0z2zunjQpEgIneaBEL1sbumoK11yJAUugfwMSCh0HyDJzuXM0mLzokFYCWIicBv510zK01JhbqICcl0ZKbZQflwOi30KfpttONlYE0KclM= ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:40:33.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 msgr2=0x7fa8500334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:33.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 0x7fa8500334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa8580096a0 tx=0x7fa858006b40).stop 2024-02-20T13:40:33.527 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 msgr2=0x7fa86c10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa860006800 tx=0x7fa860006ee0).stop 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 shutdown_connections 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa850030ff0 0x7fa8500334b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 --2- 172.21.15.158:0/346062587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa86c0275f0 0x7fa86c10cf10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 >> 172.21.15.158:0/346062587 conn(0x7fa86c0f6690 msgr2=0x7fa86c0f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 shutdown_connections 2024-02-20T13:40:33.528 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:33.536+0000 7fa872b22700 1 -- 172.21.15.158:0/346062587 wait complete. 2024-02-20T13:40:33.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:32.054538+0000 mgr.smithi158.gmoobf (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-02-20T13:40:33.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:32.162119+0000 mon.smithi158 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:33.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:32.163769+0000 mon.smithi158 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:34.134 INFO:teuthology.orchestra.run.smithi158.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:40:34.134 INFO:teuthology.orchestra.run.smithi158.stderr:Adding key to root@localhost authorized_keys... 2024-02-20T13:40:34.135 INFO:teuthology.orchestra.run.smithi158.stderr:Adding host smithi158... 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f43a08cb700 1 Processor -- start 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f43a08cb700 1 -- start start 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f43a08cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f43a08cb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0fac00 con 0x7f439c0fa2f0 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f439a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:34.652 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f439a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56640/0 (socket says 172.21.15.158:56640) 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.659+0000 7f439a59c700 1 -- 172.21.15.158:0/3059242817 learned_addr learned my addr 172.21.15.158:0/3059242817 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f439a59c700 1 -- 172.21.15.158:0/3059242817 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c0fad40 con 0x7f439c0fa2f0 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f439a59c700 1 --2- 172.21.15.158:0/3059242817 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f438c00b5f0 tx=0x7f438c006ee0).ready entity=mon.0 client_cookie=73ad27f30b80a424 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f439959a700 1 -- 172.21.15.158:0/3059242817 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f438c00d600 con 0x7f439c0fa2f0 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f439959a700 1 -- 172.21.15.158:0/3059242817 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f438c00dbc0 con 0x7f439c0fa2f0 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f439959a700 1 -- 172.21.15.158:0/3059242817 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f438c011b90 con 0x7f439c0fa2f0 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f43a08cb700 1 -- 172.21.15.158:0/3059242817 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 msgr2=0x7f439c0fa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.660+0000 7f43a08cb700 1 --2- 172.21.15.158:0/3059242817 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f438c00b5f0 tx=0x7f438c006ee0).stop 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- 172.21.15.158:0/3059242817 shutdown_connections 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 --2- 172.21.15.158:0/3059242817 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c0fa710 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- 172.21.15.158:0/3059242817 >> 172.21.15.158:0/3059242817 conn(0x7f439c0f5e10 msgr2=0x7f439c0f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- 172.21.15.158:0/3059242817 shutdown_connections 2024-02-20T13:40:34.653 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- 172.21.15.158:0/3059242817 wait complete. 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 Processor -- start 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- start start 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f43a08cb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c10b280 con 0x7f439c0fa2f0 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f439a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f439a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56646/0 (socket says 172.21.15.158:56646) 2024-02-20T13:40:34.654 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.661+0000 7f439a59c700 1 -- 172.21.15.158:0/2363236038 learned_addr learned my addr 172.21.15.158:0/2363236038 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:34.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f439a59c700 1 -- 172.21.15.158:0/2363236038 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c00b200 con 0x7f439c0fa2f0 2024-02-20T13:40:34.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f439a59c700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f438c000c00 tx=0x7f438c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:34.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f438c01b410 con 0x7f439c0fa2f0 2024-02-20T13:40:34.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f438c01b9d0 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f438c01a620 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c10b480 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.662+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c10c0c0 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.663+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f438c02b070 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.663+0000 7f438affd700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 0x7f43800334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.663+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f438c042ba0 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.663+0000 7f4399d9b700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 0x7f43800334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.663+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c10b910 con 0x7f439c0fa2f0 2024-02-20T13:40:34.656 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.664+0000 7f4399d9b700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 0x7f43800334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:34.658 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.668+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f438c01f080 con 0x7f439c0fa2f0 2024-02-20T13:40:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:34.823+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi158", "addr": "172.21.15.158", "target": ["mon-mgr", ""]}) v1 -- 0x7f439c0fde70 con 0x7f4380030ff0 2024-02-20T13:40:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Added host 'smithi158' with addr '172.21.15.158' 2024-02-20T13:40:35.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.420+0000 7f438affd700 1 -- 172.21.15.158:0/2363236038 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f439c0fde70 con 0x7f4380030ff0 2024-02-20T13:40:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 msgr2=0x7f43800334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:35.413 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 0x7f43800334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).stop 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 msgr2=0x7f439c10ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f438c000c00 tx=0x7f438c006ee0).stop 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 shutdown_connections 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f4380030ff0 0x7f43800334b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 --2- 172.21.15.158:0/2363236038 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f439c0fa2f0 0x7f439c10ad90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.422+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 >> 172.21.15.158:0/2363236038 conn(0x7f439c0f5e10 msgr2=0x7f439c0ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.423+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 shutdown_connections 2024-02-20T13:40:35.414 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:35.423+0000 7f43a08cb700 1 -- 172.21.15.158:0/2363236038 wait complete. 2024-02-20T13:40:35.434 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:33.535157+0000 mgr.smithi158.gmoobf (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:35.983 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying mon service with default placement... 2024-02-20T13:40:36.246 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:34.825533+0000 mgr.smithi158.gmoobf (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi158", "addr": "172.21.15.158", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:36.246 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:35.420208+0000 mon.smithi158 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:36.247 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:35.420712+0000 mon.smithi158 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:36.498 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.503+0000 7ff32a160700 1 Processor -- start 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.503+0000 7ff32a160700 1 -- start start 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.503+0000 7ff32a160700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.503+0000 7ff32a160700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240fca20 con 0x7ff324027630 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56650/0 (socket says 172.21.15.158:56650) 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3237fe700 1 -- 172.21.15.158:0/3736870914 learned_addr learned my addr 172.21.15.158:0/3736870914 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3237fe700 1 -- 172.21.15.158:0/3736870914 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324101520 con 0x7ff324027630 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3237fe700 1 --2- 172.21.15.158:0/3736870914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=dccd5faceb59b5d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3227fc700 1 -- 172.21.15.158:0/3736870914 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31400d600 con 0x7ff324027630 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3227fc700 1 -- 172.21.15.158:0/3736870914 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff31400dbc0 con 0x7ff324027630 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.504+0000 7ff3227fc700 1 -- 172.21.15.158:0/3736870914 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff314011b90 con 0x7ff324027630 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 -- 172.21.15.158:0/3736870914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 msgr2=0x7ff324101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:36.499 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 --2- 172.21.15.158:0/3736870914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).stop 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 -- 172.21.15.158:0/3736870914 shutdown_connections 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 --2- 172.21.15.158:0/3736870914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff324101030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 -- 172.21.15.158:0/3736870914 >> 172.21.15.158:0/3736870914 conn(0x7ff3240f6690 msgr2=0x7ff3240f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 -- 172.21.15.158:0/3736870914 shutdown_connections 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 -- 172.21.15.158:0/3736870914 wait complete. 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.505+0000 7ff32a160700 1 Processor -- start 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff32a160700 1 -- start start 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff32a160700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff32a160700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32410d3c0 con 0x7ff324027630 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56656/0 (socket says 172.21.15.158:56656) 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff3237fe700 1 -- 172.21.15.158:0/1519134961 learned_addr learned my addr 172.21.15.158:0/1519134961 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:36.500 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff3237fe700 1 -- 172.21.15.158:0/1519134961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31400b200 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff3237fe700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff314000c00 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.506+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31401b410 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.507+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32410d5c0 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.507+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff31401b9d0 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.507+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31401a620 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.507+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32410da50 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.507+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7ff31402b070 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.508+0000 7ff30bfff700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 0x7ff310033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.508+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7ff314042c60 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.508+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff324045bb0 con 0x7ff324027630 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.508+0000 7ff322ffd700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 0x7ff310033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:36.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.508+0000 7ff322ffd700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 0x7ff310033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff3180096a0 tx=0x7ff318006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:36.502 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.512+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff314016080 con 0x7ff324027630 2024-02-20T13:40:36.662 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.671+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7ff32410e370 con 0x7ff310030fa0 2024-02-20T13:40:36.666 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.675+0000 7ff30bfff700 1 -- 172.21.15.158:0/1519134961 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff32410e370 con 0x7ff310030fa0 2024-02-20T13:40:36.666 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-20T13:40:36.668 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 msgr2=0x7ff310033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 0x7ff310033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff3180096a0 tx=0x7ff318006b40).stop 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 msgr2=0x7ff32410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff314000c00 tx=0x7ff314006ee0).stop 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 shutdown_connections 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7ff310030fa0 0x7ff310033460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 --2- 172.21.15.158:0/1519134961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff324027630 0x7ff32410ced0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 >> 172.21.15.158:0/1519134961 conn(0x7ff3240f6690 msgr2=0x7ff3240f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 shutdown_connections 2024-02-20T13:40:36.669 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:36.677+0000 7ff32a160700 1 -- 172.21.15.158:0/1519134961 wait complete. 2024-02-20T13:40:37.182 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying mgr service with default placement... 2024-02-20T13:40:37.286 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:35.420489+0000 mgr.smithi158.gmoobf (mgr.14116) 9 : cephadm [INF] Added host smithi158 2024-02-20T13:40:37.286 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:36.676040+0000 mon.smithi158 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f6b1eb700 1 Processor -- start 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f6b1eb700 1 -- start start 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f6b1eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f6b1eb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640ff100 con 0x7f6f64027650 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f68f87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f68f87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56660/0 (socket says 172.21.15.158:56660) 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f68f87700 1 -- 172.21.15.158:0/4188073703 learned_addr learned my addr 172.21.15.158:0/4188073703 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.701+0000 7f6f68f87700 1 -- 172.21.15.158:0/4188073703 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640ff240 con 0x7f6f64027650 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f68f87700 1 --2- 172.21.15.158:0/4188073703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6f5800b5f0 tx=0x7f6f58006ee0).ready entity=mon.0 client_cookie=49bb513a2ab77afe server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f637fe700 1 -- 172.21.15.158:0/4188073703 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f5800d600 con 0x7f6f64027650 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f637fe700 1 -- 172.21.15.158:0/4188073703 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6f5800dbc0 con 0x7f6f64027650 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f637fe700 1 -- 172.21.15.158:0/4188073703 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f58011b90 con 0x7f6f64027650 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/4188073703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 msgr2=0x7f6f640fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/4188073703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6f5800b5f0 tx=0x7f6f58006ee0).stop 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/4188073703 shutdown_connections 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/4188073703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f640fec10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/4188073703 >> 172.21.15.158:0/4188073703 conn(0x7f6f640f6690 msgr2=0x7f6f640f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/4188073703 shutdown_connections 2024-02-20T13:40:37.694 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.702+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/4188073703 wait complete. 2024-02-20T13:40:37.696 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.703+0000 7f6f6b1eb700 1 Processor -- start 2024-02-20T13:40:37.696 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.703+0000 7f6f6b1eb700 1 -- start start 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.703+0000 7f6f6b1eb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.703+0000 7f6f6b1eb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6410b260 con 0x7f6f64027650 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f68f87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f68f87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56666/0 (socket says 172.21.15.158:56666) 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f68f87700 1 -- 172.21.15.158:0/854228342 learned_addr learned my addr 172.21.15.158:0/854228342 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f68f87700 1 -- 172.21.15.158:0/854228342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5800b200 con 0x7f6f64027650 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f68f87700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6f58006800 tx=0x7f6f58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f5800d9c0 con 0x7f6f64027650 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6f5801b870 con 0x7f6f64027650 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6410b460 con 0x7f6f64027650 2024-02-20T13:40:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f5801a740 con 0x7f6f64027650 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.704+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6410c070 con 0x7f6f64027650 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f6f5802b070 con 0x7f6f64027650 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f617fa700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 0x7f6f480334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6f58042c90 con 0x7f6f64027650 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f63fff700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 0x7f6f480334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f640591e0 con 0x7f6f64027650 2024-02-20T13:40:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.705+0000 7f6f63fff700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 0x7f6f480334b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f6f500096a0 tx=0x7f6f50006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:37.700 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.709+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6f5801f070 con 0x7f6f64027650 2024-02-20T13:40:37.860 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.869+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f64100050 con 0x7f6f48030ff0 2024-02-20T13:40:37.863 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.873+0000 7f6f617fa700 1 -- 172.21.15.158:0/854228342 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6f64100050 con 0x7f6f48030ff0 2024-02-20T13:40:37.864 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-20T13:40:37.866 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 msgr2=0x7f6f480334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:37.866 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 0x7f6f480334b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f6f500096a0 tx=0x7f6f50006b40).stop 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 msgr2=0x7f6f6410ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6f58006800 tx=0x7f6f58006ee0).stop 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 shutdown_connections 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f6f48030ff0 0x7f6f480334b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 --2- 172.21.15.158:0/854228342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f64027650 0x7f6f6410ad70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 >> 172.21.15.158:0/854228342 conn(0x7f6f640f6690 msgr2=0x7f6f640f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 shutdown_connections 2024-02-20T13:40:37.867 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:37.875+0000 7f6f6b1eb700 1 -- 172.21.15.158:0/854228342 wait complete. 2024-02-20T13:40:38.219 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:36.672898+0000 mgr.smithi158.gmoobf (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:38.219 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:36.673626+0000 mgr.smithi158.gmoobf (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-20T13:40:38.219 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:37.873643+0000 mon.smithi158 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:38.447 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying crash service with default placement... 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.041+0000 7eff6bd5c700 1 Processor -- start 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.041+0000 7eff6bd5c700 1 -- start start 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff6bd5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff6bd5c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff640fca20 con 0x7eff64027630 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff69af8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:39.038 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff69af8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51166/0 (socket says 172.21.15.158:51166) 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff69af8700 1 -- 172.21.15.158:0/448344997 learned_addr learned my addr 172.21.15.158:0/448344997 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff69af8700 1 -- 172.21.15.158:0/448344997 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64101520 con 0x7eff64027630 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff69af8700 1 --2- 172.21.15.158:0/448344997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7eff5400b5f0 tx=0x7eff54006ee0).ready entity=mon.0 client_cookie=99e34fd94d93cac4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.042+0000 7eff68af6700 1 -- 172.21.15.158:0/448344997 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5400d600 con 0x7eff64027630 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff68af6700 1 -- 172.21.15.158:0/448344997 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7eff5400dbc0 con 0x7eff64027630 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff68af6700 1 -- 172.21.15.158:0/448344997 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff54011b90 con 0x7eff64027630 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 -- 172.21.15.158:0/448344997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 msgr2=0x7eff64101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/448344997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7eff5400b5f0 tx=0x7eff54006ee0).stop 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 -- 172.21.15.158:0/448344997 shutdown_connections 2024-02-20T13:40:39.039 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/448344997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff64101030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 -- 172.21.15.158:0/448344997 >> 172.21.15.158:0/448344997 conn(0x7eff640f6690 msgr2=0x7eff640f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 -- 172.21.15.158:0/448344997 shutdown_connections 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.043+0000 7eff6bd5c700 1 -- 172.21.15.158:0/448344997 wait complete. 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff6bd5c700 1 Processor -- start 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff6bd5c700 1 -- start start 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff6bd5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff6bd5c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6410d3c0 con 0x7eff64027630 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff69af8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff69af8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51182/0 (socket says 172.21.15.158:51182) 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff69af8700 1 -- 172.21.15.158:0/717778579 learned_addr learned my addr 172.21.15.158:0/717778579 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:39.040 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff69af8700 1 -- 172.21.15.158:0/717778579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5400b200 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff69af8700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7eff54000c00 tx=0x7eff54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.044+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5401b410 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6410d5c0 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7eff5401b9d0 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5401a620 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6410da50 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7eff5402b070 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff527fc700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 0x7eff4c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.045+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff64045bb0 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.046+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7eff54042c70 con 0x7eff64027630 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.046+0000 7eff692f7700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 0x7eff4c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:39.041 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.046+0000 7eff692f7700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 0x7eff4c033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7eff600096a0 tx=0x7eff60006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:39.042 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.050+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7eff5401f080 con 0x7eff64027630 2024-02-20T13:40:39.203 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.212+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7eff6410e2b0 con 0x7eff4c030fa0 2024-02-20T13:40:39.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.216+0000 7eff527fc700 1 -- 172.21.15.158:0/717778579 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7eff6410e2b0 con 0x7eff4c030fa0 2024-02-20T13:40:39.207 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 msgr2=0x7eff4c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 0x7eff4c033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7eff600096a0 tx=0x7eff60006b40).stop 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 msgr2=0x7eff6410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7eff54000c00 tx=0x7eff54006ee0).stop 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 shutdown_connections 2024-02-20T13:40:39.209 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7eff4c030fa0 0x7eff4c033460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:39.210 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 --2- 172.21.15.158:0/717778579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff64027630 0x7eff6410ced0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:39.210 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 >> 172.21.15.158:0/717778579 conn(0x7eff640f6690 msgr2=0x7eff640f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:39.210 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 shutdown_connections 2024-02-20T13:40:39.210 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:39.218+0000 7eff6bd5c700 1 -- 172.21.15.158:0/717778579 wait complete. 2024-02-20T13:40:39.526 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:37.870962+0000 mgr.smithi158.gmoobf (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:39.526 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:37.871471+0000 mgr.smithi158.gmoobf (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-20T13:40:39.738 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying prometheus service with default placement... 2024-02-20T13:40:40.258 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d9d9d700 1 Processor -- start 2024-02-20T13:40:40.258 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d9d9d700 1 -- start start 2024-02-20T13:40:40.258 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d9d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d9d9d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40ff1a0 con 0x7f55d40275b0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51196/0 (socket says 172.21.15.158:51196) 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.264+0000 7f55d37fe700 1 -- 172.21.15.158:0/186652110 learned_addr learned my addr 172.21.15.158:0/186652110 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d37fe700 1 -- 172.21.15.158:0/186652110 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40ff2e0 con 0x7f55d40275b0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d37fe700 1 --2- 172.21.15.158:0/186652110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f55c800b5f0 tx=0x7f55c8006ee0).ready entity=mon.0 client_cookie=ac431fd824b57a53 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d27fc700 1 -- 172.21.15.158:0/186652110 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c800d600 con 0x7f55d40275b0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d27fc700 1 -- 172.21.15.158:0/186652110 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f55c800dbc0 con 0x7f55d40275b0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d27fc700 1 -- 172.21.15.158:0/186652110 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c8011b90 con 0x7f55d40275b0 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d9d9d700 1 -- 172.21.15.158:0/186652110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 msgr2=0x7f55d40fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.265+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/186652110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f55c800b5f0 tx=0x7f55c8006ee0).stop 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 -- 172.21.15.158:0/186652110 shutdown_connections 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/186652110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fecb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 -- 172.21.15.158:0/186652110 >> 172.21.15.158:0/186652110 conn(0x7f55d40f6690 msgr2=0x7f55d40f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 -- 172.21.15.158:0/186652110 shutdown_connections 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 -- 172.21.15.158:0/186652110 wait complete. 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.266+0000 7f55d9d9d700 1 Processor -- start 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d9d9d700 1 -- start start 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d9d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d9d9d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40fabb0 con 0x7f55d40275b0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51202/0 (socket says 172.21.15.158:51202) 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d37fe700 1 -- 172.21.15.158:0/725884133 learned_addr learned my addr 172.21.15.158:0/725884133 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d37fe700 1 -- 172.21.15.158:0/725884133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c800b200 con 0x7f55d40275b0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55d37fe700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f55c8000c00 tx=0x7f55c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.267+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c800de00 con 0x7f55d40275b0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d40fad50 con 0x7f55d40275b0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f55c801b870 con 0x7f55d40275b0 2024-02-20T13:40:40.260 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c801a670 con 0x7f55d40275b0 2024-02-20T13:40:40.261 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d40fb1e0 con 0x7f55d40275b0 2024-02-20T13:40:40.261 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f55c802b070 con 0x7f55d40275b0 2024-02-20T13:40:40.261 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.268+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d4045bb0 con 0x7f55d40275b0 2024-02-20T13:40:40.262 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.269+0000 7f55bbfff700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 0x7f55c0033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:40.262 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.269+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f55c8042ce0 con 0x7f55d40275b0 2024-02-20T13:40:40.262 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.269+0000 7f55d2ffd700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 0x7f55c0033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:40.262 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.269+0000 7f55d2ffd700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 0x7f55c0033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f55c40096a0 tx=0x7f55c4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:40.263 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.273+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f55c802b8f0 con 0x7f55d40275b0 2024-02-20T13:40:40.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.431+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f55d40fb420 con 0x7f55c0030fa0 2024-02-20T13:40:40.425 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.434+0000 7f55bbfff700 1 -- 172.21.15.158:0/725884133 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f55d40fb420 con 0x7f55c0030fa0 2024-02-20T13:40:40.425 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 msgr2=0x7f55c0033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 0x7f55c0033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f55c40096a0 tx=0x7f55c4006b40).stop 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 msgr2=0x7f55d40fc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f55c8000c00 tx=0x7f55c8006ee0).stop 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 shutdown_connections 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f55c0030fa0 0x7f55c0033460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 --2- 172.21.15.158:0/725884133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55d40275b0 0x7f55d40fc730 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 >> 172.21.15.158:0/725884133 conn(0x7f55d40f6690 msgr2=0x7f55d40f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 shutdown_connections 2024-02-20T13:40:40.428 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:40.436+0000 7f55d9d9d700 1 -- 172.21.15.158:0/725884133 wait complete. 2024-02-20T13:40:40.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:39.214114+0000 mgr.smithi158.gmoobf (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:40.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:39.214774+0000 mgr.smithi158.gmoobf (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-20T13:40:40.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:39.217003+0000 mon.smithi158 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:41.070 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying grafana service with default placement... 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.574+0000 7fa72213f700 1 Processor -- start 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.574+0000 7fa72213f700 1 -- start start 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa72213f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa72213f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c0ff190 con 0x7fa71c070c20 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51204/0 (socket says 172.21.15.158:51204) 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71b7fe700 1 -- 172.21.15.158:0/1028787753 learned_addr learned my addr 172.21.15.158:0/1028787753 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71b7fe700 1 -- 172.21.15.158:0/1028787753 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c0ff2d0 con 0x7fa71c070c20 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71b7fe700 1 --2- 172.21.15.158:0/1028787753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa71000b5f0 tx=0x7fa710006ee0).ready entity=mon.0 client_cookie=2eb83a3247095d52 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71a7fc700 1 -- 172.21.15.158:0/1028787753 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71000d600 con 0x7fa71c070c20 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.575+0000 7fa71a7fc700 1 -- 172.21.15.158:0/1028787753 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa71000dbc0 con 0x7fa71c070c20 2024-02-20T13:40:41.570 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa71a7fc700 1 -- 172.21.15.158:0/1028787753 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa710011b90 con 0x7fa71c070c20 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 -- 172.21.15.158:0/1028787753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 msgr2=0x7fa71c0feca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 --2- 172.21.15.158:0/1028787753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa71000b5f0 tx=0x7fa710006ee0).stop 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 -- 172.21.15.158:0/1028787753 shutdown_connections 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 --2- 172.21.15.158:0/1028787753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c0feca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 -- 172.21.15.158:0/1028787753 >> 172.21.15.158:0/1028787753 conn(0x7fa71c0f66a0 msgr2=0x7fa71c0f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 -- 172.21.15.158:0/1028787753 shutdown_connections 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.576+0000 7fa72213f700 1 -- 172.21.15.158:0/1028787753 wait complete. 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa72213f700 1 Processor -- start 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa72213f700 1 -- start start 2024-02-20T13:40:41.571 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa72213f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa72213f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c10b280 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa71b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa71b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51206/0 (socket says 172.21.15.158:51206) 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa71b7fe700 1 -- 172.21.15.158:0/138853332 learned_addr learned my addr 172.21.15.158:0/138853332 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa71b7fe700 1 -- 172.21.15.158:0/138853332 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71000b200 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.577+0000 7fa71b7fe700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa710000c00 tx=0x7fa710006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71000de00 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa71001b870 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71c10b480 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa71001a670 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71c10c0c0 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.578+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa71002b070 con 0x7fa71c070c20 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.579+0000 7fa703fff700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 0x7fa708033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:41.572 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.579+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa710042cb0 con 0x7fa71c070c20 2024-02-20T13:40:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.579+0000 7fa71affd700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 0x7fa708033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.579+0000 7fa71affd700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 0x7fa708033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa70c0096a0 tx=0x7fa70c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:41.573 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.579+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71c10b910 con 0x7fa71c070c20 2024-02-20T13:40:41.574 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.583+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa71001f080 con 0x7fa71c070c20 2024-02-20T13:40:41.733 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.742+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa71c1000e0 con 0x7fa708030fa0 2024-02-20T13:40:41.736 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.746+0000 7fa703fff700 1 -- 172.21.15.158:0/138853332 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa70c0092a0 con 0x7fa708030fa0 2024-02-20T13:40:41.736 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 msgr2=0x7fa708033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 0x7fa708033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa70c0096a0 tx=0x7fa70c006b40).stop 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 msgr2=0x7fa71c10ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa710000c00 tx=0x7fa710006ee0).stop 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 shutdown_connections 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fa708030fa0 0x7fa708033460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 --2- 172.21.15.158:0/138853332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa71c070c20 0x7fa71c10ad90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 >> 172.21.15.158:0/138853332 conn(0x7fa71c0f66a0 msgr2=0x7fa71c0f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:41.739 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 shutdown_connections 2024-02-20T13:40:41.740 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:41.748+0000 7fa72213f700 1 -- 172.21.15.158:0/138853332 wait complete. 2024-02-20T13:40:41.761 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:40.432555+0000 mgr.smithi158.gmoobf (mgr.14116) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:41.762 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:40.433061+0000 mgr.smithi158.gmoobf (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-20T13:40:41.762 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:40.435297+0000 mon.smithi158 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:42.360 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying node-exporter service with default placement... 2024-02-20T13:40:42.958 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.886+0000 7f75cb322700 1 Processor -- start 2024-02-20T13:40:42.958 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75cb322700 1 -- start start 2024-02-20T13:40:42.958 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75cb322700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:42.958 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75cb322700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40fca20 con 0x7f75c4027630 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75c90be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75c90be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51212/0 (socket says 172.21.15.158:51212) 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75c90be700 1 -- 172.21.15.158:0/483058019 learned_addr learned my addr 172.21.15.158:0/483058019 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75c90be700 1 -- 172.21.15.158:0/483058019 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c4101520 con 0x7f75c4027630 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75c90be700 1 --2- 172.21.15.158:0/483058019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f75b800b5f0 tx=0x7f75b8006ee0).ready entity=mon.0 client_cookie=94f0923b2860da5f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.887+0000 7f75b7fff700 1 -- 172.21.15.158:0/483058019 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b800d600 con 0x7f75c4027630 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75b7fff700 1 -- 172.21.15.158:0/483058019 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f75b800dbc0 con 0x7f75c4027630 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75b7fff700 1 -- 172.21.15.158:0/483058019 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b8011b90 con 0x7f75c4027630 2024-02-20T13:40:42.959 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 -- 172.21.15.158:0/483058019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 msgr2=0x7f75c4101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 --2- 172.21.15.158:0/483058019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f75b800b5f0 tx=0x7f75b8006ee0).stop 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 -- 172.21.15.158:0/483058019 shutdown_connections 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 --2- 172.21.15.158:0/483058019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c4101030 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 -- 172.21.15.158:0/483058019 >> 172.21.15.158:0/483058019 conn(0x7f75c40f6690 msgr2=0x7f75c40f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 -- 172.21.15.158:0/483058019 shutdown_connections 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.888+0000 7f75cb322700 1 -- 172.21.15.158:0/483058019 wait complete. 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75cb322700 1 Processor -- start 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75cb322700 1 -- start start 2024-02-20T13:40:42.960 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75cb322700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75cb322700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c410d3c0 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75c90be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75c90be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51226/0 (socket says 172.21.15.158:51226) 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75c90be700 1 -- 172.21.15.158:0/1885408753 learned_addr learned my addr 172.21.15.158:0/1885408753 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75c90be700 1 -- 172.21.15.158:0/1885408753 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b800b200 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75c90be700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f75b8006800 tx=0x7f75b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b800de50 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f75b801b870 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c410d5c0 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.889+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b801a7a0 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.890+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c410da50 con 0x7f75c4027630 2024-02-20T13:40:42.961 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.890+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f75b800f070 con 0x7f75c4027630 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.890+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4045bb0 con 0x7f75c4027630 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.891+0000 7f75b5ffb700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 0x7f75ac0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.891+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f75b8042bb0 con 0x7f75c4027630 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.891+0000 7f75c88bd700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 0x7f75ac0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.891+0000 7f75c88bd700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 0x7f75ac0334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f75c00096a0 tx=0x7f75c0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:42.962 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:42.895+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f75b801b410 con 0x7f75c4027630 2024-02-20T13:40:43.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:41.743840+0000 mgr.smithi158.gmoobf (mgr.14116) 18 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:43.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:41.744390+0000 mgr.smithi158.gmoobf (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-20T13:40:43.038 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:41.746802+0000 mon.smithi158 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:43.049 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.058+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c410e370 con 0x7f75ac030ff0 2024-02-20T13:40:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.062+0000 7f75b5ffb700 1 -- 172.21.15.158:0/1885408753 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f75c410e370 con 0x7f75ac030ff0 2024-02-20T13:40:43.053 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-20T13:40:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 msgr2=0x7f75ac0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 0x7f75ac0334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f75c00096a0 tx=0x7f75c0006b40).stop 2024-02-20T13:40:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 msgr2=0x7f75c410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:43.055 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f75b8006800 tx=0x7f75b8006ee0).stop 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 shutdown_connections 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f75ac030ff0 0x7f75ac0334b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 --2- 172.21.15.158:0/1885408753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75c4027630 0x7f75c410ced0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 >> 172.21.15.158:0/1885408753 conn(0x7f75c40f6690 msgr2=0x7f75c40f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 shutdown_connections 2024-02-20T13:40:43.056 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:43.064+0000 7f75cb322700 1 -- 172.21.15.158:0/1885408753 wait complete. 2024-02-20T13:40:43.711 INFO:teuthology.orchestra.run.smithi158.stderr:Deploying alertmanager service with default placement... 2024-02-20T13:40:44.317 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:44 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:43.060344+0000 mgr.smithi158.gmoobf (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:44.318 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:44 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:43.060918+0000 mgr.smithi158.gmoobf (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-20T13:40:44.318 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:44 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:43.063033+0000 mon.smithi158 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:44.551 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64df7a5700 1 Processor -- start 2024-02-20T13:40:44.551 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64df7a5700 1 -- start start 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64df7a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64df7a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80fca40 con 0x7f64d8027610 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64dd541700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64dd541700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51242/0 (socket says 172.21.15.158:51242) 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.556+0000 7f64dd541700 1 -- 172.21.15.158:0/670252313 learned_addr learned my addr 172.21.15.158:0/670252313 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.557+0000 7f64dd541700 1 -- 172.21.15.158:0/670252313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d8101540 con 0x7f64d8027610 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.557+0000 7f64dd541700 1 --2- 172.21.15.158:0/670252313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f64c800b5f0 tx=0x7f64c8006ee0).ready entity=mon.0 client_cookie=755354040e1615bf server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.557+0000 7f64c7fff700 1 -- 172.21.15.158:0/670252313 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c800d600 con 0x7f64d8027610 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.557+0000 7f64c7fff700 1 -- 172.21.15.158:0/670252313 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f64c800dbc0 con 0x7f64d8027610 2024-02-20T13:40:44.552 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.557+0000 7f64c7fff700 1 -- 172.21.15.158:0/670252313 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c8011b90 con 0x7f64d8027610 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- 172.21.15.158:0/670252313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 msgr2=0x7f64d8101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 --2- 172.21.15.158:0/670252313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f64c800b5f0 tx=0x7f64c8006ee0).stop 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- 172.21.15.158:0/670252313 shutdown_connections 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 --2- 172.21.15.158:0/670252313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d8101050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- 172.21.15.158:0/670252313 >> 172.21.15.158:0/670252313 conn(0x7f64d80f6690 msgr2=0x7f64d80f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- 172.21.15.158:0/670252313 shutdown_connections 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- 172.21.15.158:0/670252313 wait complete. 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 Processor -- start 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- start start 2024-02-20T13:40:44.553 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:44.554 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.558+0000 7f64df7a5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d810d3c0 con 0x7f64d8027610 2024-02-20T13:40:44.554 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64dd541700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:44.554 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64dd541700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51256/0 (socket says 172.21.15.158:51256) 2024-02-20T13:40:44.554 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64dd541700 1 -- 172.21.15.158:0/2969324202 learned_addr learned my addr 172.21.15.158:0/2969324202 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:44.554 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64dd541700 1 -- 172.21.15.158:0/2969324202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c800b200 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64dd541700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f64c8006800 tx=0x7f64c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c800de50 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f64c801b870 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d810d5c0 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c801a7a0 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.559+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d810da50 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.560+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f64c800f070 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.560+0000 7f64c5ffb700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 0x7f64c00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.560+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f64c8042be0 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.561+0000 7f64dcd40700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 0x7f64c00334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.561+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d8045bb0 con 0x7f64d8027610 2024-02-20T13:40:44.555 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.561+0000 7f64dcd40700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 0x7f64c00334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f64d40096a0 tx=0x7f64d4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:44.556 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.565+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f64c8016030 con 0x7f64d8027610 2024-02-20T13:40:44.723 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.732+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f64d810e370 con 0x7f64c0030ff0 2024-02-20T13:40:44.727 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.737+0000 7f64c5ffb700 1 -- 172.21.15.158:0/2969324202 <== mgr.14116 v2:172.21.15.158:6800/1783554617 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f64d810e370 con 0x7f64c0030ff0 2024-02-20T13:40:44.728 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-20T13:40:44.731 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 msgr2=0x7f64c00334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:44.731 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 0x7f64c00334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f64d40096a0 tx=0x7f64d4006b40).stop 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 msgr2=0x7f64d810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f64c8006800 tx=0x7f64c8006ee0).stop 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 shutdown_connections 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f64c0030ff0 0x7f64c00334b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 --2- 172.21.15.158:0/2969324202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64d8027610 0x7f64d810ced0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 >> 172.21.15.158:0/2969324202 conn(0x7f64d80f6690 msgr2=0x7f64d80f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:44.733 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 shutdown_connections 2024-02-20T13:40:44.733 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:44.739+0000 7f64df7a5700 1 -- 172.21.15.158:0/2969324202 wait complete. 2024-02-20T13:40:46.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:44.734096+0000 mgr.smithi158.gmoobf (mgr.14116) 22 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:40:46.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:40:44.734617+0000 mgr.smithi158.gmoobf (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-20T13:40:46.037 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:44.737546+0000 mon.smithi158 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558ddb3700 1 Processor -- start 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558ddb3700 1 -- start start 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558ddb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558ddb3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880fb690 con 0x7f55880fad80 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558cdb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558cdb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51262/0 (socket says 172.21.15.158:51262) 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.310+0000 7f558cdb1700 1 -- 172.21.15.158:0/3064524325 learned_addr learned my addr 172.21.15.158:0/3064524325 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.311+0000 7f558cdb1700 1 -- 172.21.15.158:0/3064524325 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880fb7d0 con 0x7f55880fad80 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.311+0000 7f558cdb1700 1 --2- 172.21.15.158:0/3064524325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f557c00b5f0 tx=0x7f557c006ee0).ready entity=mon.0 client_cookie=f24ed775630f3326 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:46.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.311+0000 7f55877fe700 1 -- 172.21.15.158:0/3064524325 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c00d600 con 0x7f55880fad80 2024-02-20T13:40:46.303 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.311+0000 7f55877fe700 1 -- 172.21.15.158:0/3064524325 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f557c00dbf0 con 0x7f55880fad80 2024-02-20T13:40:46.303 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.311+0000 7f55877fe700 1 -- 172.21.15.158:0/3064524325 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c011bf0 con 0x7f55880fad80 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 -- 172.21.15.158:0/3064524325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 msgr2=0x7f55880fb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 --2- 172.21.15.158:0/3064524325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f557c00b5f0 tx=0x7f557c006ee0).stop 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 -- 172.21.15.158:0/3064524325 shutdown_connections 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 --2- 172.21.15.158:0/3064524325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f55880fb1a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 -- 172.21.15.158:0/3064524325 >> 172.21.15.158:0/3064524325 conn(0x7f55880f64e0 msgr2=0x7f55880f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 -- 172.21.15.158:0/3064524325 shutdown_connections 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.312+0000 7f558ddb3700 1 -- 172.21.15.158:0/3064524325 wait complete. 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558ddb3700 1 Processor -- start 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558ddb3700 1 -- start start 2024-02-20T13:40:46.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558ddb3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558ddb3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558806adf0 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558cdb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558cdb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51268/0 (socket says 172.21.15.158:51268) 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.313+0000 7f558cdb1700 1 -- 172.21.15.158:0/1357342574 learned_addr learned my addr 172.21.15.158:0/1357342574 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f558cdb1700 1 -- 172.21.15.158:0/1357342574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c00b200 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f558cdb1700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f557c011e30 tx=0x7f557c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c01b410 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558806aff0 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f557c01ba00 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c01a750 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.314+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558810df10 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.315+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f557c02b070 con 0x7f55880fad80 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.315+0000 7f55857fa700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 0x7f556c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:46.307 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.315+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f557c043ab0 con 0x7f55880fad80 2024-02-20T13:40:46.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.315+0000 7f5587fff700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 0x7f556c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:46.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.315+0000 7f5587fff700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 0x7f556c033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f55740096a0 tx=0x7f5574006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:46.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.316+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5588045bb0 con 0x7f55880fad80 2024-02-20T13:40:46.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.320+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f557c016080 con 0x7f55880fad80 2024-02-20T13:40:46.485 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.494+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f55880fcaf0 con 0x7f55880fad80 2024-02-20T13:40:46.488 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.497+0000 7f55857fa700 1 -- 172.21.15.158:0/1357342574 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f557c01f070 con 0x7f55880fad80 2024-02-20T13:40:46.491 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 msgr2=0x7f556c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:46.491 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 0x7f556c033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f55740096a0 tx=0x7f5574006b40).stop 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 msgr2=0x7f558810d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f557c011e30 tx=0x7f557c006ee0).stop 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 shutdown_connections 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f556c030fa0 0x7f556c033460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 --2- 172.21.15.158:0/1357342574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55880fad80 0x7f558810d860 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 >> 172.21.15.158:0/1357342574 conn(0x7f55880f64e0 msgr2=0x7f55880f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 shutdown_connections 2024-02-20T13:40:46.492 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:46.500+0000 7f558ddb3700 1 -- 172.21.15.158:0/1357342574 wait complete. 2024-02-20T13:40:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:46.127477+0000 mon.smithi158 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:46.362496+0000 mon.smithi158 (mon.0) 65 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:46.498371+0000 mon.smithi158 (mon.0) 66 : audit [INF] from='client.? 172.21.15.158:0/1357342574' entity='client.admin' 2024-02-20T13:40:47.600 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe41c2d700 1 Processor -- start 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe41c2d700 1 -- start start 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe41c2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe41c2d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c072090 con 0x7fbe3c071c70 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe40c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe40c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51276/0 (socket says 172.21.15.158:51276) 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe40c2b700 1 -- 172.21.15.158:0/3055202037 learned_addr learned my addr 172.21.15.158:0/3055202037 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe40c2b700 1 -- 172.21.15.158:0/3055202037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c070660 con 0x7fbe3c071c70 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.607+0000 7fbe40c2b700 1 --2- 172.21.15.158:0/3055202037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbe3000b5f0 tx=0x7fbe30006ee0).ready entity=mon.0 client_cookie=b97c4067b2e2a184 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe3b7fe700 1 -- 172.21.15.158:0/3055202037 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3000d600 con 0x7fbe3c071c70 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe3b7fe700 1 -- 172.21.15.158:0/3055202037 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe3000dbf0 con 0x7fbe3c071c70 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe3b7fe700 1 -- 172.21.15.158:0/3055202037 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe30011bf0 con 0x7fbe3c071c70 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3055202037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 msgr2=0x7fbe3c070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3055202037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbe3000b5f0 tx=0x7fbe30006ee0).stop 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3055202037 shutdown_connections 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3055202037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c070170 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3055202037 >> 172.21.15.158:0/3055202037 conn(0x7fbe3c0f7610 msgr2=0x7fbe3c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3055202037 shutdown_connections 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.608+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3055202037 wait complete. 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe41c2d700 1 Processor -- start 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe41c2d700 1 -- start start 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe41c2d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe41c2d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c10d280 con 0x7fbe3c071c70 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe40c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe40c2b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51288/0 (socket says 172.21.15.158:51288) 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe40c2b700 1 -- 172.21.15.158:0/3566647450 learned_addr learned my addr 172.21.15.158:0/3566647450 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.609+0000 7fbe40c2b700 1 -- 172.21.15.158:0/3566647450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3000b200 con 0x7fbe3c071c70 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe40c2b700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbe3000bdc0 tx=0x7fbe30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:47.602 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3001b410 con 0x7fbe3c071c70 2024-02-20T13:40:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe3001ba00 con 0x7fbe3c071c70 2024-02-20T13:40:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3c10d480 con 0x7fbe3c071c70 2024-02-20T13:40:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3001a6b0 con 0x7fbe3c071c70 2024-02-20T13:40:47.603 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.610+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3c10e0c0 con 0x7fbe3c071c70 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fbe3002b070 con 0x7fbe3c071c70 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe397fa700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 0x7fbe200334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3c10d910 con 0x7fbe3c071c70 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fbe30042bd0 con 0x7fbe3c071c70 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe3bfff700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 0x7fbe200334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:47.604 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.611+0000 7fbe3bfff700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 0x7fbe200334b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:47.605 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.615+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbe30042e60 con 0x7fbe3c071c70 2024-02-20T13:40:47.764 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.773+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fbe3c0591e0 con 0x7fbe3c071c70 2024-02-20T13:40:47.766 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.776+0000 7fbe397fa700 1 -- 172.21.15.158:0/3566647450 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fbe3003a070 con 0x7fbe3c071c70 2024-02-20T13:40:47.769 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 msgr2=0x7fbe200334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:47.769 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 0x7fbe200334b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).stop 2024-02-20T13:40:47.769 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 msgr2=0x7fbe3c10cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbe3000bdc0 tx=0x7fbe30006ee0).stop 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 shutdown_connections 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7fbe20030ff0 0x7fbe200334b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 --2- 172.21.15.158:0/3566647450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe3c071c70 0x7fbe3c10cd90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.778+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 >> 172.21.15.158:0/3566647450 conn(0x7fbe3c0f7610 msgr2=0x7fbe3c0f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.779+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 shutdown_connections 2024-02-20T13:40:47.770 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:47.779+0000 7fbe41c2d700 1 -- 172.21.15.158:0/3566647450 wait complete. 2024-02-20T13:40:48.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:46.365484+0000 mgr.smithi158.gmoobf (mgr.14116) 24 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:40:48.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:47.650151+0000 mon.smithi158 (mon.0) 67 : audit [INF] from='mgr.14116 172.21.15.158:0/2852244995' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:48.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:47.776944+0000 mon.smithi158 (mon.0) 68 : audit [INF] from='client.? 172.21.15.158:0/3566647450' entity='client.admin' 2024-02-20T13:40:48.553 INFO:teuthology.orchestra.run.smithi158.stderr:Enabling the dashboard module... 2024-02-20T13:40:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.419+0000 7f22e4a18700 1 Processor -- start 2024-02-20T13:40:49.411 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.419+0000 7f22e4a18700 1 -- start start 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22e4a18700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22e4a18700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00ff0e0 con 0x7f22e006ae50 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22de59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22de59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46392/0 (socket says 172.21.15.158:46392) 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22de59c700 1 -- 172.21.15.158:0/3204156783 learned_addr learned my addr 172.21.15.158:0/3204156783 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22de59c700 1 -- 172.21.15.158:0/3204156783 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00ff220 con 0x7f22e006ae50 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22de59c700 1 --2- 172.21.15.158:0/3204156783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f22d400b5f0 tx=0x7f22d4006ee0).ready entity=mon.0 client_cookie=7d12ce56c707f2b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22dd59a700 1 -- 172.21.15.158:0/3204156783 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d400d600 con 0x7f22e006ae50 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.420+0000 7f22dd59a700 1 -- 172.21.15.158:0/3204156783 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22d400dbf0 con 0x7f22e006ae50 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22dd59a700 1 -- 172.21.15.158:0/3204156783 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d4011bf0 con 0x7f22e006ae50 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 -- 172.21.15.158:0/3204156783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 msgr2=0x7f22e00febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:49.412 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 --2- 172.21.15.158:0/3204156783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f22d400b5f0 tx=0x7f22d4006ee0).stop 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 -- 172.21.15.158:0/3204156783 shutdown_connections 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 --2- 172.21.15.158:0/3204156783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e00febf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 -- 172.21.15.158:0/3204156783 >> 172.21.15.158:0/3204156783 conn(0x7f22e00f6690 msgr2=0x7f22e00f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 -- 172.21.15.158:0/3204156783 shutdown_connections 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.421+0000 7f22e4a18700 1 -- 172.21.15.158:0/3204156783 wait complete. 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22e4a18700 1 Processor -- start 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22e4a18700 1 -- start start 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22e4a18700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22e4a18700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010b270 con 0x7f22e006ae50 2024-02-20T13:40:49.415 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22de59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22de59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46398/0 (socket says 172.21.15.158:46398) 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.422+0000 7f22de59c700 1 -- 172.21.15.158:0/2298790244 learned_addr learned my addr 172.21.15.158:0/2298790244 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22de59c700 1 -- 172.21.15.158:0/2298790244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d400b200 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22de59c700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f22d4006800 tx=0x7f22d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d400dec0 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22d401b8a0 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e010b470 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d401a7b0 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.423+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e010c080 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.424+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f22d402b070 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.424+0000 7f22ceffd700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 0x7f22c4033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.424+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f22d4042c30 con 0x7f22e006ae50 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.424+0000 7f22ddd9b700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 0x7f22c4033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.424+0000 7f22ddd9b700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 0x7f22c4033460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f22d00096a0 tx=0x7f22d0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:49.416 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.425+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e00591e0 con 0x7f22e006ae50 2024-02-20T13:40:49.419 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.429+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f22d4016070 con 0x7f22e006ae50 2024-02-20T13:40:49.615 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:49.625+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f22e00fff80 con 0x7f22e006ae50 2024-02-20T13:40:50.121 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.130+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f22d400f070 con 0x7f22e006ae50 2024-02-20T13:40:50.122 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.131+0000 7f22ceffd700 1 -- 172.21.15.158:0/2298790244 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f22d402c780 con 0x7f22e006ae50 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 msgr2=0x7f22c4033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 0x7f22c4033460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f22d00096a0 tx=0x7f22d0006b40).stop 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 msgr2=0x7f22e010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f22d4006800 tx=0x7f22d4006ee0).stop 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 shutdown_connections 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f22c4030fa0 0x7f22c4033460 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 --2- 172.21.15.158:0/2298790244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22e006ae50 0x7f22e010ad80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 >> 172.21.15.158:0/2298790244 conn(0x7f22e00f6690 msgr2=0x7f22e00f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 shutdown_connections 2024-02-20T13:40:50.125 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:50.133+0000 7f22e4a18700 1 -- 172.21.15.158:0/2298790244 wait complete. 2024-02-20T13:40:50.484 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:48.365769+0000 mgr.smithi158.gmoobf (mgr.14116) 25 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:40:50.484 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:49.626097+0000 mon.smithi158 (mon.0) 69 : audit [INF] from='client.? 172.21.15.158:0/2298790244' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T13:40:51.401 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.376+0000 7f332a6da700 1 Processor -- start 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.376+0000 7f332a6da700 1 -- start start 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.376+0000 7f332a6da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.376+0000 7f332a6da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3324069920 con 0x7f3324069010 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.377+0000 7f3323fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.377+0000 7f3323fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46404/0 (socket says 172.21.15.158:46404) 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.377+0000 7f3323fff700 1 -- 172.21.15.158:0/3704845587 learned_addr learned my addr 172.21.15.158:0/3704845587 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:51.421 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.377+0000 7f3323fff700 1 -- 172.21.15.158:0/3704845587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324069a60 con 0x7f3324069010 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f3323fff700 1 --2- 172.21.15.158:0/3704845587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f331c008960 tx=0x7f331c008350).ready entity=mon.0 client_cookie=e295e557b3b3cc59 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f3322ffd700 1 -- 172.21.15.158:0/3704845587 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f331c00dd40 con 0x7f3324069010 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f3322ffd700 1 -- 172.21.15.158:0/3704845587 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f331c0148a0 con 0x7f3324069010 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f3322ffd700 1 -- 172.21.15.158:0/3704845587 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f331c013710 con 0x7f3324069010 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f332a6da700 1 -- 172.21.15.158:0/3704845587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 msgr2=0x7f3324069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f332a6da700 1 --2- 172.21.15.158:0/3704845587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f331c008960 tx=0x7f331c008350).stop 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f332a6da700 1 -- 172.21.15.158:0/3704845587 shutdown_connections 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f332a6da700 1 --2- 172.21.15.158:0/3704845587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324069430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.378+0000 7f332a6da700 1 -- 172.21.15.158:0/3704845587 >> 172.21.15.158:0/3704845587 conn(0x7f3324063480 msgr2=0x7f33240658a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.379+0000 7f332a6da700 1 -- 172.21.15.158:0/3704845587 shutdown_connections 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.379+0000 7f332a6da700 1 -- 172.21.15.158:0/3704845587 wait complete. 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.379+0000 7f332a6da700 1 Processor -- start 2024-02-20T13:40:51.422 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.379+0000 7f332a6da700 1 -- start start 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f332a6da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f332a6da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3324111770 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3323fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3323fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46406/0 (socket says 172.21.15.158:46406) 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3323fff700 1 -- 172.21.15.158:0/4111035482 learned_addr learned my addr 172.21.15.158:0/4111035482 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3323fff700 1 -- 172.21.15.158:0/4111035482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0085f0 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3323fff700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f331c006f00 tx=0x7f331c008350).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.380+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f331c013900 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.381+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f331c013ef0 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.381+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f331c01d3a0 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.381+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3324111970 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.381+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3324111e00 con 0x7f3324069010 2024-02-20T13:40:51.423 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.382+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3324045bb0 con 0x7f3324069010 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.382+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f331c013a60 con 0x7f3324069010 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.382+0000 7f3320ff9700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 0x7f3310033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.383+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f331c045b80 con 0x7f3324069010 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.383+0000 7f33237fe700 1 -- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 msgr2=0x7f3310033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.383+0000 7f33237fe700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 0x7f3310033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:40:51.424 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.387+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f331c02ebb0 con 0x7f3324069010 2024-02-20T13:40:51.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:50.132348+0000 mon.smithi158 (mon.0) 70 : audit [INF] from='client.? 172.21.15.158:0/2298790244' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T13:40:51.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:50.132399+0000 mon.smithi158 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi158.gmoobf(active, since 23s) 2024-02-20T13:40:51.575 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.584+0000 7f33237fe700 1 -- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 msgr2=0x7f3310033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:51.575 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.584+0000 7f33237fe700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 0x7f3310033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:40:51.593 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.598+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3324112e30 con 0x7f3324069010 2024-02-20T13:40:51.593 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.598+0000 7f3320ff9700 1 -- 172.21.15.158:0/4111035482 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f331c019030 con 0x7f3324069010 2024-02-20T13:40:51.593 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:51.593 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-20T13:40:51.593 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:40:51.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "active_name": "smithi158.gmoobf", 2024-02-20T13:40:51.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T13:40:51.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:51.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 msgr2=0x7f3310033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:40:51.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 0x7f3310033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:51.595 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 msgr2=0x7f3324111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:51.595 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f331c006f00 tx=0x7f331c008350).stop 2024-02-20T13:40:51.595 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 shutdown_connections 2024-02-20T13:40:51.595 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f3310031040 0x7f3310033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:51.595 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 --2- 172.21.15.158:0/4111035482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3324069010 0x7f3324111280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:51.596 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.600+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 >> 172.21.15.158:0/4111035482 conn(0x7f3324063480 msgr2=0x7f3324105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:51.596 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.601+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 shutdown_connections 2024-02-20T13:40:51.596 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:51.601+0000 7f332a6da700 1 -- 172.21.15.158:0/4111035482 wait complete. 2024-02-20T13:40:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for the mgr to restart... 2024-02-20T13:40:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:Waiting for mgr epoch 9... 2024-02-20T13:40:52.263 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:52 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:51.599134+0000 mon.smithi158 (mon.0) 72 : audit [DBG] from='client.? 172.21.15.158:0/4111035482' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86f0e38700 1 Processor -- start 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86f0e38700 1 -- start start 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86f0e38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86f0e38700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec0fca80 con 0x7f86ec06adb0 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86ea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86ea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46412/0 (socket says 172.21.15.158:46412) 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.904+0000 7f86ea59c700 1 -- 172.21.15.158:0/3475518404 learned_addr learned my addr 172.21.15.158:0/3475518404 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:52.898 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86ea59c700 1 -- 172.21.15.158:0/3475518404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec101580 con 0x7f86ec06adb0 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86ea59c700 1 --2- 172.21.15.158:0/3475518404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f86dc00b5f0 tx=0x7f86dc006ee0).ready entity=mon.0 client_cookie=8ac3b796c3876ee2 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86e959a700 1 -- 172.21.15.158:0/3475518404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86dc00d600 con 0x7f86ec06adb0 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86e959a700 1 -- 172.21.15.158:0/3475518404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f86dc00dbf0 con 0x7f86ec06adb0 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86e959a700 1 -- 172.21.15.158:0/3475518404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86dc011bf0 con 0x7f86ec06adb0 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86f0e38700 1 -- 172.21.15.158:0/3475518404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 msgr2=0x7f86ec101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.905+0000 7f86f0e38700 1 --2- 172.21.15.158:0/3475518404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f86dc00b5f0 tx=0x7f86dc006ee0).stop 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 -- 172.21.15.158:0/3475518404 shutdown_connections 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 --2- 172.21.15.158:0/3475518404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec101090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 -- 172.21.15.158:0/3475518404 >> 172.21.15.158:0/3475518404 conn(0x7f86ec0f6690 msgr2=0x7f86ec0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 -- 172.21.15.158:0/3475518404 shutdown_connections 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 -- 172.21.15.158:0/3475518404 wait complete. 2024-02-20T13:40:52.899 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 Processor -- start 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.906+0000 7f86f0e38700 1 -- start start 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86f0e38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86f0e38700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec10d470 con 0x7f86ec06adb0 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86ea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86ea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:46424/0 (socket says 172.21.15.158:46424) 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86ea59c700 1 -- 172.21.15.158:0/2394675835 learned_addr learned my addr 172.21.15.158:0/2394675835 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86ea59c700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc00b200 con 0x7f86ec06adb0 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.907+0000 7f86ea59c700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f86dc006800 tx=0x7f86dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.908+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86dc00dec0 con 0x7f86ec06adb0 2024-02-20T13:40:52.901 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.908+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec10d670 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.908+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f86dc01b8a0 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.908+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86dc01a6f0 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.908+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec10db00 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f86dc02b070 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86daffd700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f86dc042bc0 con 0x7f86ec06adb0 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86e9d9b700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 msgr2=0x7f86d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:40:52.902 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:52.909+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86ec0591e0 con 0x7f86d0030ff0 2024-02-20T13:40:53.100 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:53.110+0000 7f86e9d9b700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 msgr2=0x7f86d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:53.100 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:53.110+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:40:53.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:53.511+0000 7f86e9d9b700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 msgr2=0x7f86d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:53.501 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:53.511+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T13:40:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:54.312+0000 7f86e9d9b700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 msgr2=0x7f86d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.158:6800/1783554617 2024-02-20T13:40:54.303 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:54.312+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T13:40:55.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:55.319+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7f86dc00f070 con 0x7f86ec06adb0 2024-02-20T13:40:55.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:55.319+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 msgr2=0x7f86d00334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:40:55.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:55.319+0000 7f86daffd700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:56.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.320+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f86dc024400 con 0x7f86ec06adb0 2024-02-20T13:40:56.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.320+0000 7f86daffd700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 0x7f86d0036d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:56.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.320+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86ec0591e0 con 0x7f86d0034990 2024-02-20T13:40:56.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.320+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 0x7f86d0036d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:56.311 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.321+0000 7f86e9d9b700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 0x7f86d0036d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f86e00036d0 tx=0x7f86e0006ce0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:56.312 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.321+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f86ec0591e0 con 0x7f86d0034990 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.327+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f86ec045bb0 con 0x7f86d0034990 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.327+0000 7f86daffd700 1 -- 172.21.15.158:0/2394675835 <== mgr.14156 v2:172.21.15.158:6800/770403169 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f86ec045bb0 con 0x7f86d0034990 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: { 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: } 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 msgr2=0x7f86d0036d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 0x7f86d0036d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f86e00036d0 tx=0x7f86e0006ce0).stop 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 msgr2=0x7f86ec10cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f86dc006800 tx=0x7f86dc006ee0).stop 2024-02-20T13:40:56.319 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 shutdown_connections 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f86d0034990 0x7f86d0036d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86ec06adb0 0x7f86ec10cf80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 --2- 172.21.15.158:0/2394675835 >> [v2:172.21.15.158:6800/1783554617,v1:172.21.15.158:6801/1783554617] conn(0x7f86d0030ff0 0x7f86d00334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 >> 172.21.15.158:0/2394675835 conn(0x7f86ec0f6690 msgr2=0x7f86ec0f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 shutdown_connections 2024-02-20T13:40:56.320 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:56.328+0000 7f86f0e38700 1 -- 172.21.15.158:0/2394675835 wait complete. 2024-02-20T13:40:56.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:55.267918+0000 mon.smithi158 (mon.0) 73 : cluster [INF] Active manager daemon smithi158.gmoobf restarted 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:55.268376+0000 mon.smithi158 (mon.0) 74 : cluster [INF] Activating manager daemon smithi158.gmoobf 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:55.270020+0000 mon.smithi158 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:55.320847+0000 mon.smithi158 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi158.gmoobf(active, starting, since 0.0525974s) 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.321640+0000 mon.smithi158 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata", "id": "smithi158"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.322166+0000 mon.smithi158 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mgr metadata", "who": "smithi158.gmoobf", "id": "smithi158.gmoobf"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.322817+0000 mon.smithi158 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.322969+0000 mon.smithi158 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.323109+0000 mon.smithi158 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:55.336141+0000 mon.smithi158 (mon.0) 82 : cluster [INF] Manager daemon smithi158.gmoobf is now available 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.433671+0000 mon.smithi158 (mon.0) 83 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.435943+0000 mon.smithi158 (mon.0) 84 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:40:56.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.457176+0000 mon.smithi158 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:40:56.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:55.458641+0000 mon.smithi158 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:40:56.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:56.044479+0000 mon.smithi158 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:40:57.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:56.322191+0000 mgr.smithi158.gmoobf (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T13:40:57.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:56.322875+0000 mon.smithi158 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi158.gmoobf(active, since 1.05462s) 2024-02-20T13:40:57.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:56.328509+0000 mgr.smithi158.gmoobf (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T13:40:57.885 INFO:teuthology.orchestra.run.smithi158.stderr:mgr epoch 9 is available 2024-02-20T13:40:57.885 INFO:teuthology.orchestra.run.smithi158.stderr:Generating a dashboard self-signed certificate... 2024-02-20T13:40:58.455 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.460+0000 7f225630c700 1 Processor -- start 2024-02-20T13:40:58.455 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.460+0000 7f225630c700 1 -- start start 2024-02-20T13:40:58.455 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.460+0000 7f225630c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:58.455 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.460+0000 7f225630c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22500fb7f0 con 0x7f22500faee0 2024-02-20T13:40:58.455 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56350/0 (socket says 172.21.15.158:56350) 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224ffff700 1 -- 172.21.15.158:0/4059162710 learned_addr learned my addr 172.21.15.158:0/4059162710 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224ffff700 1 -- 172.21.15.158:0/4059162710 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22500fb930 con 0x7f22500faee0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224ffff700 1 --2- 172.21.15.158:0/4059162710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f224000b5f0 tx=0x7f2240006ee0).ready entity=mon.0 client_cookie=a59a9b74fc4085e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224effd700 1 -- 172.21.15.158:0/4059162710 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f224000d600 con 0x7f22500faee0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224effd700 1 -- 172.21.15.158:0/4059162710 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f224000dbf0 con 0x7f22500faee0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.461+0000 7f224effd700 1 -- 172.21.15.158:0/4059162710 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2240011bf0 con 0x7f22500faee0 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 -- 172.21.15.158:0/4059162710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 msgr2=0x7f22500fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 --2- 172.21.15.158:0/4059162710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f224000b5f0 tx=0x7f2240006ee0).stop 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 -- 172.21.15.158:0/4059162710 shutdown_connections 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 --2- 172.21.15.158:0/4059162710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f22500fb300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 -- 172.21.15.158:0/4059162710 >> 172.21.15.158:0/4059162710 conn(0x7f22500f6690 msgr2=0x7f22500f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 -- 172.21.15.158:0/4059162710 shutdown_connections 2024-02-20T13:40:58.456 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 -- 172.21.15.158:0/4059162710 wait complete. 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.462+0000 7f225630c700 1 Processor -- start 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f225630c700 1 -- start start 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f225630c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f225630c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2250027650 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f224ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f224ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56362/0 (socket says 172.21.15.158:56362) 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f224ffff700 1 -- 172.21.15.158:0/2190338143 learned_addr learned my addr 172.21.15.158:0/2190338143 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f224ffff700 1 -- 172.21.15.158:0/2190338143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224000b200 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.463+0000 7f224ffff700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2240011de0 tx=0x7f2240006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.464+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f224001b410 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.464+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f224001ba00 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.464+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f224001a770 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.464+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2250027850 con 0x7f22500faee0 2024-02-20T13:40:58.457 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.464+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225010e110 con 0x7f22500faee0 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f224002b070 con 0x7f22500faee0 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f224cff9700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 0x7f223c033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2240042c20 con 0x7f22500faee0 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f224f7fe700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 0x7f223c033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f224f7fe700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 0x7f223c033500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f22440096a0 tx=0x7f2244006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:58.458 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.465+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2250045bb0 con 0x7f22500faee0 2024-02-20T13:40:58.460 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.469+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f224001f020 con 0x7f22500faee0 2024-02-20T13:40:58.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:40:57.324745+0000 mon.smithi158 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi158.gmoobf(active, since 2s) 2024-02-20T13:40:58.616 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.625+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f22500fccf0 con 0x7f223c031040 2024-02-20T13:40:58.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.664+0000 7f224cff9700 1 -- 172.21.15.158:0/2190338143 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f22500fccf0 con 0x7f223c031040 2024-02-20T13:40:58.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 msgr2=0x7f223c033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 0x7f223c033500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f22440096a0 tx=0x7f2244006b40).stop 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 msgr2=0x7f225010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2240011de0 tx=0x7f2240006ee0).stop 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 shutdown_connections 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f223c031040 0x7f223c033500 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:58.657 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 --2- 172.21.15.158:0/2190338143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22500faee0 0x7f225010d9e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:58.658 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.666+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 >> 172.21.15.158:0/2190338143 conn(0x7f22500f6690 msgr2=0x7f22500f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:58.658 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.667+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 shutdown_connections 2024-02-20T13:40:58.658 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:58.667+0000 7f225630c700 1 -- 172.21.15.158:0/2190338143 wait complete. 2024-02-20T13:40:59.334 INFO:teuthology.orchestra.run.smithi158.stderr:Creating initial admin user... 2024-02-20T13:40:59.916 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc855d6700 1 Processor -- start 2024-02-20T13:40:59.916 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc855d6700 1 -- start start 2024-02-20T13:40:59.916 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc855d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:59.916 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc855d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc800fdf60 con 0x7fdc800fd650 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56376/0 (socket says 172.21.15.158:56376) 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.923+0000 7fdc7effd700 1 -- 172.21.15.158:0/3273897052 learned_addr learned my addr 172.21.15.158:0/3273897052 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc7effd700 1 -- 172.21.15.158:0/3273897052 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc800fe0a0 con 0x7fdc800fd650 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc7effd700 1 --2- 172.21.15.158:0/3273897052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdc70009960 tx=0x7fdc700092d0).ready entity=mon.0 client_cookie=25195557afff1b25 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc7dffb700 1 -- 172.21.15.158:0/3273897052 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7000b9c0 con 0x7fdc800fd650 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc7dffb700 1 -- 172.21.15.158:0/3273897052 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdc70011400 con 0x7fdc800fd650 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc7dffb700 1 -- 172.21.15.158:0/3273897052 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc70010580 con 0x7fdc800fd650 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 -- 172.21.15.158:0/3273897052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 msgr2=0x7fdc800fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 --2- 172.21.15.158:0/3273897052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdc70009960 tx=0x7fdc700092d0).stop 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 -- 172.21.15.158:0/3273897052 shutdown_connections 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 --2- 172.21.15.158:0/3273897052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fda70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 -- 172.21.15.158:0/3273897052 >> 172.21.15.158:0/3273897052 conn(0x7fdc800f5e30 msgr2=0x7fdc800f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 -- 172.21.15.158:0/3273897052 shutdown_connections 2024-02-20T13:40:59.917 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.924+0000 7fdc855d6700 1 -- 172.21.15.158:0/3273897052 wait complete. 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc855d6700 1 Processor -- start 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc855d6700 1 -- start start 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc855d6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc855d6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc800fb3a0 con 0x7fdc800fd650 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56386/0 (socket says 172.21.15.158:56386) 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.925+0000 7fdc7effd700 1 -- 172.21.15.158:0/2644314606 learned_addr learned my addr 172.21.15.158:0/2644314606 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc7effd700 1 -- 172.21.15.158:0/2644314606 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70009570 con 0x7fdc800fd650 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc7effd700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdc70011cb0 tx=0x7fdc700092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc70010840 con 0x7fdc800fd650 2024-02-20T13:40:59.918 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdc70010e30 con 0x7fdc800fd650 2024-02-20T13:40:59.919 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc800fb540 con 0x7fdc800fd650 2024-02-20T13:40:59.919 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7001bc80 con 0x7fdc800fd650 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.926+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc800fb9d0 con 0x7fdc800fd650 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.927+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fdc7000f070 con 0x7fdc800fd650 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.927+0000 7fdc677fe700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 0x7fdc6c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.927+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdc700430e0 con 0x7fdc800fd650 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.927+0000 7fdc7e7fc700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 0x7fdc6c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.927+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc800591e0 con 0x7fdc800fd650 2024-02-20T13:40:59.920 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.928+0000 7fdc7e7fc700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 0x7fdc6c0334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdc740096a0 tx=0x7fdc74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:40:59.921 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:40:59.931+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdc70024400 con 0x7fdc800fd650 2024-02-20T13:41:00.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:58.626957+0000 mgr.smithi158.gmoobf (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:41:00.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:58.663559+0000 mon.smithi158 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:00.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:40:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:40:58.665311+0000 mon.smithi158 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:00.089 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.099+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdc80045bb0 con 0x7fdc6c030ff0 2024-02-20T13:41:00.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.315+0000 7fdc677fe700 1 -- 172.21.15.158:0/2644314606 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fdc80045bb0 con 0x7fdc6c030ff0 2024-02-20T13:41:00.306 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$QztZwHKGgtBUoEenDRAJleoNNXus3hrnDmH5ll2eawcwv3Un9ptCy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708436460, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T13:41:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 msgr2=0x7fdc6c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 0x7fdc6c0334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdc740096a0 tx=0x7fdc74006b40).stop 2024-02-20T13:41:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 msgr2=0x7fdc800fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:00.308 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdc70011cb0 tx=0x7fdc700092d0).stop 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 shutdown_connections 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdc6c030ff0 0x7fdc6c0334b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 --2- 172.21.15.158:0/2644314606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc800fd650 0x7fdc800fcf20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 >> 172.21.15.158:0/2644314606 conn(0x7fdc800f5e30 msgr2=0x7fdc800ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 shutdown_connections 2024-02-20T13:41:00.309 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:00.317+0000 7fdc855d6700 1 -- 172.21.15.158:0/2644314606 wait complete. 2024-02-20T13:41:00.995 INFO:teuthology.orchestra.run.smithi158.stderr:Fetching dashboard port number... 2024-02-20T13:41:00.997 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:00 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:00.100580+0000 mgr.smithi158.gmoobf (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:41:00.997 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:00 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:00.316410+0000 mon.smithi158 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:01.578 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.585+0000 7f480428e700 1 Processor -- start 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.585+0000 7f480428e700 1 -- start start 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480428e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480428e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0fca60 con 0x7f47fc06add0 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480202a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480202a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56402/0 (socket says 172.21.15.158:56402) 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480202a700 1 -- 172.21.15.158:0/1741464736 learned_addr learned my addr 172.21.15.158:0/1741464736 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480202a700 1 -- 172.21.15.158:0/1741464736 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc101560 con 0x7f47fc06add0 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f480202a700 1 --2- 172.21.15.158:0/1741464736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f47ec00b5f0 tx=0x7f47ec006ee0).ready entity=mon.0 client_cookie=774be069cfc6df41 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f4801028700 1 -- 172.21.15.158:0/1741464736 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec00d600 con 0x7f47fc06add0 2024-02-20T13:41:01.579 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f4801028700 1 -- 172.21.15.158:0/1741464736 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f47ec00dbf0 con 0x7f47fc06add0 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.586+0000 7f4801028700 1 -- 172.21.15.158:0/1741464736 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec011bf0 con 0x7f47fc06add0 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 -- 172.21.15.158:0/1741464736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 msgr2=0x7f47fc101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 --2- 172.21.15.158:0/1741464736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f47ec00b5f0 tx=0x7f47ec006ee0).stop 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 -- 172.21.15.158:0/1741464736 shutdown_connections 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 --2- 172.21.15.158:0/1741464736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc101070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 -- 172.21.15.158:0/1741464736 >> 172.21.15.158:0/1741464736 conn(0x7f47fc0f6690 msgr2=0x7f47fc0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 -- 172.21.15.158:0/1741464736 shutdown_connections 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.587+0000 7f480428e700 1 -- 172.21.15.158:0/1741464736 wait complete. 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480428e700 1 Processor -- start 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480428e700 1 -- start start 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480428e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:01.580 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480428e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc10d430 con 0x7f47fc06add0 2024-02-20T13:41:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480202a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480202a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56408/0 (socket says 172.21.15.158:56408) 2024-02-20T13:41:01.581 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480202a700 1 -- 172.21.15.158:0/1341983881 learned_addr learned my addr 172.21.15.158:0/1341983881 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.588+0000 7f480202a700 1 -- 172.21.15.158:0/1341983881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec00b200 con 0x7f47fc06add0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f480202a700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f47ec006800 tx=0x7f47ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec00dec0 con 0x7f47fc06add0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f47ec01b8a0 con 0x7f47fc06add0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc10d630 con 0x7f47fc06add0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec01a6f0 con 0x7f47fc06add0 2024-02-20T13:41:01.582 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.589+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc10dac0 con 0x7f47fc06add0 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f47ec02b070 con 0x7f47fc06add0 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f47eaffd700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 0x7f47e4033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47ec042a80 con 0x7f47fc06add0 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc045bb0 con 0x7f47fc06add0 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f4801829700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 0x7f47e4033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:01.583 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.590+0000 7f4801829700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 0x7f47e4033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f47f80096a0 tx=0x7f47f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:01.584 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.594+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f47ec016070 con 0x7f47fc06add0 2024-02-20T13:41:01.655 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.664+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f47ec00f070 con 0x7f47fc06add0 2024-02-20T13:41:01.735 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.744+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f47fc102100 con 0x7f47fc06add0 2024-02-20T13:41:01.735 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.745+0000 7f47eaffd700 1 -- 172.21.15.158:0/1341983881 <== mon.0 v2:172.21.15.158:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f47ec01f070 con 0x7f47fc06add0 2024-02-20T13:41:01.735 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 8443 2024-02-20T13:41:01.737 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 msgr2=0x7f47e4033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:01.737 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 0x7f47e4033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f47f80096a0 tx=0x7f47f8006b40).stop 2024-02-20T13:41:01.737 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 msgr2=0x7f47fc10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:01.737 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f47ec006800 tx=0x7f47ec006ee0).stop 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 shutdown_connections 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47e4031040 0x7f47e4033500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 --2- 172.21.15.158:0/1341983881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47fc06add0 0x7f47fc10cf40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 >> 172.21.15.158:0/1341983881 conn(0x7f47fc0f6690 msgr2=0x7f47fc0f80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 shutdown_connections 2024-02-20T13:41:01.738 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:01.746+0000 7f480428e700 1 -- 172.21.15.158:0/1341983881 wait complete. 2024-02-20T13:41:02.369 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: disabled 2024-02-20T13:41:02.439 INFO:teuthology.orchestra.run.smithi158.stderr:systemctl: inactive 2024-02-20T13:41:02.439 INFO:teuthology.orchestra.run.smithi158.stderr:firewalld.service is not enabled 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr:Ceph Dashboard is now available at: 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr: URL: https://smithi158.front.sepia.ceph.com:8443/ 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr: User: admin 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr: Password: dmis2n3jsx 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:02.440 INFO:teuthology.orchestra.run.smithi158.stderr:Enabling autotune for osd_memory_target 2024-02-20T13:41:02.966 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.970+0000 7fdff359e700 1 Processor -- start 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff359e700 1 -- start start 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff359e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec0fef30 con 0x7fdfec06ae50 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56410/0 (socket says 172.21.15.158:56410) 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff259c700 1 -- 172.21.15.158:0/2920911083 learned_addr learned my addr 172.21.15.158:0/2920911083 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff259c700 1 -- 172.21.15.158:0/2920911083 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec0ff070 con 0x7fdfec06ae50 2024-02-20T13:41:02.967 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.971+0000 7fdff259c700 1 --2- 172.21.15.158:0/2920911083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdfe400b5f0 tx=0x7fdfe4006ee0).ready entity=mon.0 client_cookie=b8f5538000a76de9 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff159a700 1 -- 172.21.15.158:0/2920911083 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe400d600 con 0x7fdfec06ae50 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff159a700 1 -- 172.21.15.158:0/2920911083 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdfe400dbf0 con 0x7fdfec06ae50 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff159a700 1 -- 172.21.15.158:0/2920911083 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe4011bf0 con 0x7fdfec06ae50 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 -- 172.21.15.158:0/2920911083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 msgr2=0x7fdfec0fea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 --2- 172.21.15.158:0/2920911083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdfe400b5f0 tx=0x7fdfe4006ee0).stop 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 -- 172.21.15.158:0/2920911083 shutdown_connections 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 --2- 172.21.15.158:0/2920911083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec0fea40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 -- 172.21.15.158:0/2920911083 >> 172.21.15.158:0/2920911083 conn(0x7fdfec0f64e0 msgr2=0x7fdfec0f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 -- 172.21.15.158:0/2920911083 shutdown_connections 2024-02-20T13:41:02.968 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.972+0000 7fdff359e700 1 -- 172.21.15.158:0/2920911083 wait complete. 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff359e700 1 Processor -- start 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff359e700 1 -- start start 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff359e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec10b040 con 0x7fdfec06ae50 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56422/0 (socket says 172.21.15.158:56422) 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff259c700 1 -- 172.21.15.158:0/3616407965 learned_addr learned my addr 172.21.15.158:0/3616407965 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff259c700 1 -- 172.21.15.158:0/3616407965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe400b200 con 0x7fdfec06ae50 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdff259c700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fdfe4011de0 tx=0x7fdfe4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.973+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe401b410 con 0x7fdfec06ae50 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.974+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdfe401ba00 con 0x7fdfec06ae50 2024-02-20T13:41:02.969 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.974+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec10b240 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.974+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe401a770 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.974+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec10b6d0 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.974+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdfe402b070 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.975+0000 7fdfdeffd700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 0x7fdfd4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.975+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec045bb0 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.975+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdfe4042d70 con 0x7fdfec06ae50 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.975+0000 7fdff1d9b700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 0x7fdfd4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:02.970 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.975+0000 7fdff1d9b700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 0x7fdfd4033550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:02.971 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:02.979+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdfe401f020 con 0x7fdfec06ae50 2024-02-20T13:41:03.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:01.666590+0000 mon.smithi158 (mon.0) 93 : cluster [DBG] mgrmap e13: smithi158.gmoobf(active, since 6s) 2024-02-20T13:41:03.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:01.745640+0000 mon.smithi158 (mon.0) 94 : audit [DBG] from='client.? 172.21.15.158:0/1341983881' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T13:41:03.121 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.130+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fdfec0ffd30 con 0x7fdfec06ae50 2024-02-20T13:41:03.121 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.130+0000 7fdfdeffd700 1 -- 172.21.15.158:0/3616407965 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7fdfe4016070 con 0x7fdfec06ae50 2024-02-20T13:41:03.123 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 msgr2=0x7fdfd4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:03.123 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 0x7fdfd4033550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).stop 2024-02-20T13:41:03.123 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 msgr2=0x7fdfec10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fdfe4011de0 tx=0x7fdfe4006ee0).stop 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 shutdown_connections 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdfd4031090 0x7fdfd4033550 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 --2- 172.21.15.158:0/3616407965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfec06ae50 0x7fdfec10ab50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 >> 172.21.15.158:0/3616407965 conn(0x7fdfec0f64e0 msgr2=0x7fdfec0f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 shutdown_connections 2024-02-20T13:41:03.124 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:03.132+0000 7fdff359e700 1 -- 172.21.15.158:0/3616407965 wait complete. 2024-02-20T13:41:04.353 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f16534f6700 1 Processor -- start 2024-02-20T13:41:04.354 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f16534f6700 1 -- start start 2024-02-20T13:41:04.355 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f16534f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:04.355 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f16534f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c0fca40 con 0x7f164c06adf0 2024-02-20T13:41:04.355 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f1651292700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:04.355 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f1651292700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56436/0 (socket says 172.21.15.158:56436) 2024-02-20T13:41:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.361+0000 7f1651292700 1 -- 172.21.15.158:0/3012712461 learned_addr learned my addr 172.21.15.158:0/3012712461 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f1651292700 1 -- 172.21.15.158:0/3012712461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c101540 con 0x7f164c06adf0 2024-02-20T13:41:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f1651292700 1 --2- 172.21.15.158:0/3012712461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f164800b5f0 tx=0x7f1648006ee0).ready entity=mon.0 client_cookie=1596eaca77c5d5d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:04.356 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f163bfff700 1 -- 172.21.15.158:0/3012712461 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164800d600 con 0x7f164c06adf0 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f163bfff700 1 -- 172.21.15.158:0/3012712461 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f164800dbf0 con 0x7f164c06adf0 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f163bfff700 1 -- 172.21.15.158:0/3012712461 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648011bf0 con 0x7f164c06adf0 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 -- 172.21.15.158:0/3012712461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 msgr2=0x7f164c101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 --2- 172.21.15.158:0/3012712461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f164800b5f0 tx=0x7f1648006ee0).stop 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 -- 172.21.15.158:0/3012712461 shutdown_connections 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 --2- 172.21.15.158:0/3012712461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c101050 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 -- 172.21.15.158:0/3012712461 >> 172.21.15.158:0/3012712461 conn(0x7f164c0f6690 msgr2=0x7f164c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 -- 172.21.15.158:0/3012712461 shutdown_connections 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.362+0000 7f16534f6700 1 -- 172.21.15.158:0/3012712461 wait complete. 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f16534f6700 1 Processor -- start 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f16534f6700 1 -- start start 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f16534f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f16534f6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c10d410 con 0x7f164c06adf0 2024-02-20T13:41:04.357 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f1651292700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f1651292700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56442/0 (socket says 172.21.15.158:56442) 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f1651292700 1 -- 172.21.15.158:0/1042945239 learned_addr learned my addr 172.21.15.158:0/1042945239 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.363+0000 7f1651292700 1 -- 172.21.15.158:0/1042945239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164800b200 con 0x7f164c06adf0 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f1651292700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1648011de0 tx=0x7f1648006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164801b410 con 0x7f164c06adf0 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c10d610 con 0x7f164c06adf0 2024-02-20T13:41:04.358 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f164801ba00 con 0x7f164c06adf0 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164801a770 con 0x7f164c06adf0 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.364+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c10daa0 con 0x7f164c06adf0 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f164802b070 con 0x7f164c06adf0 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f1639ffb700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 0x7f1634033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f164801bb70 con 0x7f164c06adf0 2024-02-20T13:41:04.359 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f1650a91700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 0x7f1634033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c045bb0 con 0x7f164c06adf0 2024-02-20T13:41:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.365+0000 7f1650a91700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 0x7f1634033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f163c0096a0 tx=0x7f163c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:04.360 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.369+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1648016070 con 0x7f164c06adf0 2024-02-20T13:41:04.592 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.602+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f164c10efb0 con 0x7f164c06adf0 2024-02-20T13:41:04.594 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.604+0000 7f1639ffb700 1 -- 172.21.15.158:0/1042945239 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v31)=0 set mgr/dashboard/cluster/status v31) v1 ==== 153+0+0 (secure 0 0 0) 0x7f16480330c0 con 0x7f164c06adf0 2024-02-20T13:41:04.597 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 msgr2=0x7f1634033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:04.597 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 0x7f1634033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f163c0096a0 tx=0x7f163c006b40).stop 2024-02-20T13:41:04.597 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 msgr2=0x7f164c10cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1648011de0 tx=0x7f1648006ee0).stop 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 shutdown_connections 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1634031090 0x7f1634033550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 --2- 172.21.15.158:0/1042945239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f164c06adf0 0x7f164c10cf20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 >> 172.21.15.158:0/1042945239 conn(0x7f164c0f6690 msgr2=0x7f164c0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 shutdown_connections 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: 2024-02-20T13:41:04.606+0000 7f16534f6700 1 -- 172.21.15.158:0/1042945239 wait complete. 2024-02-20T13:41:04.598 INFO:teuthology.orchestra.run.smithi158.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-20T13:41:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:Or, if you are only running a single cluster on this host: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: ceph telemetry on 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:For more information see: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.261 INFO:teuthology.orchestra.run.smithi158.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-20T13:41:05.262 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:41:05.262 INFO:teuthology.orchestra.run.smithi158.stderr:Bootstrap complete. 2024-02-20T13:41:05.296 INFO:tasks.cephadm:Fetching config... 2024-02-20T13:41:05.296 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:41:05.296 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T13:41:05.355 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T13:41:05.355 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:41:05.355 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T13:41:05.413 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T13:41:05.413 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:41:05.413 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/mon.smithi158/keyring of=/dev/stdout 2024-02-20T13:41:05.482 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T13:41:05.482 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-20T13:41:05.482 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T13:41:05.539 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T13:41:05.540 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDawOa5jyiIAx5VzXIF3AJIxD5o4vTDY0KOcaIz7eNkEo7x/Q9GKaR0jUNx6V8rGAYlWEPBRaH5nnkuVJbKj2LwP+jCT3O9OIlk75hIkEdLb/NrKUoTOqtJMWjIbUUyJpEVEJy3eiHKuAWcXOYTG9EO7hm90hJVMzfFch2B4w7V4Uc56RxRJIpAzO9sWxqqHeX8hUIJgspnNPjso3gPVcXTT8v0z/3dOjy5D48urL3NeQY+Bc18BrDGWItUX6OC5d7fmLfhD4BZGIAe5Jym3XgnE9YIqxE3lwFDlTWyYMt9IUYj57fMj59qslGRpYDWBnVRjRYRBcE90kQm2DcHFby0vetRL2t+554Ch1fyGMRG0frLUYck0ZdTNum9YgK8zNLi2ozBuS5fr+HJzDMlBnkxa0z2zunjQpEgIneaBEL1sbumoK11yJAUugfwMSCh0HyDJzuXM0mLzokFYCWIicBv510zK01JhbqICcl0ZKbZQflwOi30KfpttONlYE0KclM= ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T13:41:05.623 INFO:teuthology.orchestra.run.smithi158.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDawOa5jyiIAx5VzXIF3AJIxD5o4vTDY0KOcaIz7eNkEo7x/Q9GKaR0jUNx6V8rGAYlWEPBRaH5nnkuVJbKj2LwP+jCT3O9OIlk75hIkEdLb/NrKUoTOqtJMWjIbUUyJpEVEJy3eiHKuAWcXOYTG9EO7hm90hJVMzfFch2B4w7V4Uc56RxRJIpAzO9sWxqqHeX8hUIJgspnNPjso3gPVcXTT8v0z/3dOjy5D48urL3NeQY+Bc18BrDGWItUX6OC5d7fmLfhD4BZGIAe5Jym3XgnE9YIqxE3lwFDlTWyYMt9IUYj57fMj59qslGRpYDWBnVRjRYRBcE90kQm2DcHFby0vetRL2t+554Ch1fyGMRG0frLUYck0ZdTNum9YgK8zNLi2ozBuS5fr+HJzDMlBnkxa0z2zunjQpEgIneaBEL1sbumoK11yJAUugfwMSCh0HyDJzuXM0mLzokFYCWIicBv510zK01JhbqICcl0ZKbZQflwOi30KfpttONlYE0KclM= ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:41:05.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDawOa5jyiIAx5VzXIF3AJIxD5o4vTDY0KOcaIz7eNkEo7x/Q9GKaR0jUNx6V8rGAYlWEPBRaH5nnkuVJbKj2LwP+jCT3O9OIlk75hIkEdLb/NrKUoTOqtJMWjIbUUyJpEVEJy3eiHKuAWcXOYTG9EO7hm90hJVMzfFch2B4w7V4Uc56RxRJIpAzO9sWxqqHeX8hUIJgspnNPjso3gPVcXTT8v0z/3dOjy5D48urL3NeQY+Bc18BrDGWItUX6OC5d7fmLfhD4BZGIAe5Jym3XgnE9YIqxE3lwFDlTWyYMt9IUYj57fMj59qslGRpYDWBnVRjRYRBcE90kQm2DcHFby0vetRL2t+554Ch1fyGMRG0frLUYck0ZdTNum9YgK8zNLi2ozBuS5fr+HJzDMlBnkxa0z2zunjQpEgIneaBEL1sbumoK11yJAUugfwMSCh0HyDJzuXM0mLzokFYCWIicBv510zK01JhbqICcl0ZKbZQflwOi30KfpttONlYE0KclM= ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T13:41:05.678 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDawOa5jyiIAx5VzXIF3AJIxD5o4vTDY0KOcaIz7eNkEo7x/Q9GKaR0jUNx6V8rGAYlWEPBRaH5nnkuVJbKj2LwP+jCT3O9OIlk75hIkEdLb/NrKUoTOqtJMWjIbUUyJpEVEJy3eiHKuAWcXOYTG9EO7hm90hJVMzfFch2B4w7V4Uc56RxRJIpAzO9sWxqqHeX8hUIJgspnNPjso3gPVcXTT8v0z/3dOjy5D48urL3NeQY+Bc18BrDGWItUX6OC5d7fmLfhD4BZGIAe5Jym3XgnE9YIqxE3lwFDlTWyYMt9IUYj57fMj59qslGRpYDWBnVRjRYRBcE90kQm2DcHFby0vetRL2t+554Ch1fyGMRG0frLUYck0ZdTNum9YgK8zNLi2ozBuS5fr+HJzDMlBnkxa0z2zunjQpEgIneaBEL1sbumoK11yJAUugfwMSCh0HyDJzuXM0mLzokFYCWIicBv510zK01JhbqICcl0ZKbZQflwOi30KfpttONlYE0KclM= ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715 2024-02-20T13:41:05.691 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T13:41:06.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:04.604987+0000 mon.smithi158 (mon.0) 95 : audit [INF] from='client.? 172.21.15.158:0/1042945239' entity='client.admin' 2024-02-20T13:41:08.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- 172.21.15.158:0/1294235707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 msgr2=0x7faad80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 --2- 172.21.15.158:0/1294235707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad80f58d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7faad0009960 tx=0x7faad0004d10).stop 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- 172.21.15.158:0/1294235707 shutdown_connections 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 --2- 172.21.15.158:0/1294235707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad80f58d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- 172.21.15.158:0/1294235707 >> 172.21.15.158:0/1294235707 conn(0x7faad80f0c10 msgr2=0x7faad80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- 172.21.15.158:0/1294235707 shutdown_connections 2024-02-20T13:41:08.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- 172.21.15.158:0/1294235707 wait complete. 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 Processor -- start 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 -- start start 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.218+0000 7faadd0fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faadd0fe700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8060300 con 0x7faad80f54b0 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faad6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faad6d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:56460/0 (socket says 172.21.15.158:56460) 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faad6d9d700 1 -- 172.21.15.158:0/4021895377 learned_addr learned my addr 172.21.15.158:0/4021895377 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faad6d9d700 1 -- 172.21.15.158:0/4021895377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0009640 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faad6d9d700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faad0009610 tx=0x7faad0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad0018070 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8060500 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faad000b7e0 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad0013420 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.219+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8109b80 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7faad0013650 con 0x7faad80f54b0 2024-02-20T13:41:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad80591e0 con 0x7faad80f54b0 2024-02-20T13:41:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faacf7fe700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 0x7faac4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faad0042ad0 con 0x7faad80f54b0 2024-02-20T13:41:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faad659c700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 0x7faac4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.220+0000 7faad659c700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 0x7faac4033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7faac00096a0 tx=0x7faac0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:08.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.224+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faad001bd40 con 0x7faad80f54b0 2024-02-20T13:41:08.380 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.389+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7faad80fafa0 con 0x7faad80f54b0 2024-02-20T13:41:08.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.393+0000 7faacf7fe700 1 -- 172.21.15.158:0/4021895377 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7faad001bb60 con 0x7faad80f54b0 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 msgr2=0x7faac4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 0x7faac4033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7faac00096a0 tx=0x7faac0006b40).stop 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 msgr2=0x7faad81094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faad0009610 tx=0x7faad0004d10).stop 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 shutdown_connections 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faac4031090 0x7faac4033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 --2- 172.21.15.158:0/4021895377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faad80f54b0 0x7faad81094d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.394+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 >> 172.21.15.158:0/4021895377 conn(0x7faad80f0c10 msgr2=0x7faad80f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.395+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 shutdown_connections 2024-02-20T13:41:08.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:08.395+0000 7faadd0fe700 1 -- 172.21.15.158:0/4021895377 wait complete. 2024-02-20T13:41:09.312 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-20T13:41:09.312 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-20T13:41:09.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:08.262820+0000 mon.smithi158 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:09.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:08.393670+0000 mon.smithi158 (mon.0) 97 : audit [INF] from='client.? 172.21.15.158:0/4021895377' entity='client.admin' 2024-02-20T13:41:09.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:08.549883+0000 mon.smithi158 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-20T13:41:09.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:08.552805+0000 mon.smithi158 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:09.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:41:08.559665+0000 mgr.smithi158.gmoobf (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi158 on smithi158 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 -- 172.21.15.158:0/4068139268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 msgr2=0x7f66280f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 --2- 172.21.15.158:0/4068139268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66280f58d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f661c009fb0 tx=0x7f661c009b30).stop 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 -- 172.21.15.158:0/4068139268 shutdown_connections 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 --2- 172.21.15.158:0/4068139268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66280f58d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 -- 172.21.15.158:0/4068139268 >> 172.21.15.158:0/4068139268 conn(0x7f66280f0c10 msgr2=0x7f66280f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 -- 172.21.15.158:0/4068139268 shutdown_connections 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.799+0000 7f662d2b5700 1 -- 172.21.15.158:0/4068139268 wait complete. 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 Processor -- start 2024-02-20T13:41:11.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 -- start start 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628060300 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f6626ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f6626ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37872/0 (socket says 172.21.15.158:37872) 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f6626ffd700 1 -- 172.21.15.158:0/567242267 learned_addr learned my addr 172.21.15.158:0/567242267 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f6626ffd700 1 -- 172.21.15.158:0/567242267 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c005040 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f6626ffd700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f661c0089f0 tx=0x7f661c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c009320 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f661c013ec0 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6628060500 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c014d30 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.800+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6628109b70 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.801+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f661c0263a0 con 0x7f66280f54b0 2024-02-20T13:41:11.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.801+0000 7f660f7fe700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 0x7f6610033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:11.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.801+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f661c0423c0 con 0x7f66280f54b0 2024-02-20T13:41:11.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.801+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66280591e0 con 0x7f66280f54b0 2024-02-20T13:41:11.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.801+0000 7f66267fc700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 0x7f6610033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:11.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.802+0000 7f66267fc700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 0x7f6610033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f66180096a0 tx=0x7f6618006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:11.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.806+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f661c0266d0 con 0x7f66280f54b0 2024-02-20T13:41:11.950 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.960+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f66280f1d80 con 0x7f6610031090 2024-02-20T13:41:11.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.963+0000 7f660f7fe700 1 -- 172.21.15.158:0/567242267 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f66280f1d80 con 0x7f6610031090 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 msgr2=0x7f6610033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 0x7f6610033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f66180096a0 tx=0x7f6618006b40).stop 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 msgr2=0x7f66281094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f661c0089f0 tx=0x7f661c00b040).stop 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 shutdown_connections 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6610031090 0x7f6610033550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 --2- 172.21.15.158:0/567242267 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66280f54b0 0x7f66281094c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 >> 172.21.15.158:0/567242267 conn(0x7f66280f0c10 msgr2=0x7f66280f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 shutdown_connections 2024-02-20T13:41:11.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:11.965+0000 7f662d2b5700 1 -- 172.21.15.158:0/567242267 wait complete. 2024-02-20T13:41:12.539 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-02-20T13:41:12.539 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:41:12.539 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-02-20T13:41:12.561 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-20T13:41:12.561 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T13:41:12.623 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-02-20T13:41:12.623 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph orch host add smithi192 2024-02-20T13:41:13.267 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:11.961913+0000 mgr.smithi158.gmoobf (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:41:13.267 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:11.964239+0000 mon.smithi158 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:14.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- 172.21.15.158:0/3221569350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 msgr2=0x7f17400f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:14.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 --2- 172.21.15.158:0/3221569350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17400f69b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1734004760 tx=0x7f1734009ad0).stop 2024-02-20T13:41:14.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- 172.21.15.158:0/3221569350 shutdown_connections 2024-02-20T13:41:14.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 --2- 172.21.15.158:0/3221569350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17400f69b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- 172.21.15.158:0/3221569350 >> 172.21.15.158:0/3221569350 conn(0x7f17400f1cd0 msgr2=0x7f17400f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- 172.21.15.158:0/3221569350 shutdown_connections 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- 172.21.15.158:0/3221569350 wait complete. 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 Processor -- start 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 -- start start 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.335+0000 7f1744d70700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f1744d70700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740106cb0 con 0x7f17400f6590 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f173e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f173e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37890/0 (socket says 172.21.15.158:37890) 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f173e59c700 1 -- 172.21.15.158:0/888151573 learned_addr learned my addr 172.21.15.158:0/888151573 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f173e59c700 1 -- 172.21.15.158:0/888151573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734009d70 con 0x7f17400f6590 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f173e59c700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1734004170 tx=0x7f1734009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173400e040 con 0x7f17400f6590 2024-02-20T13:41:14.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1734008ef0 con 0x7f17400f6590 2024-02-20T13:41:14.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1740106eb0 con 0x7f17400f6590 2024-02-20T13:41:14.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1734014c90 con 0x7f17400f6590 2024-02-20T13:41:14.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.336+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1740107340 con 0x7f17400f6590 2024-02-20T13:41:14.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f17340283f0 con 0x7f17400f6590 2024-02-20T13:41:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f172effd700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 0x7f1728033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1740045bb0 con 0x7f17400f6590 2024-02-20T13:41:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1734042210 con 0x7f17400f6590 2024-02-20T13:41:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f173dd9b700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 0x7f1728033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.337+0000 7f173dd9b700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 0x7f1728033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f17300096a0 tx=0x7f1730006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:14.331 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.341+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1734026300 con 0x7f17400f6590 2024-02-20T13:41:14.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:14.495+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f17401083f0 con 0x7f1728031090 2024-02-20T13:41:15.099 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.109+0000 7f172effd700 1 -- 172.21.15.158:0/888151573 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f17401083f0 con 0x7f1728031090 2024-02-20T13:41:15.100 INFO:teuthology.orchestra.run.smithi158.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 msgr2=0x7f1728033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 0x7f1728033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f17300096a0 tx=0x7f1730006b40).stop 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 msgr2=0x7f17401067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1734004170 tx=0x7f1734009ad0).stop 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 shutdown_connections 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1728031090 0x7f1728033550 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 --2- 172.21.15.158:0/888151573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17400f6590 0x7f17401067c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 >> 172.21.15.158:0/888151573 conn(0x7f17400f1cd0 msgr2=0x7f17400f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 shutdown_connections 2024-02-20T13:41:15.101 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:15.111+0000 7f1744d70700 1 -- 172.21.15.158:0/888151573 wait complete. 2024-02-20T13:41:15.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T13:41:16.367 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:14.496686+0000 mgr.smithi158.gmoobf (mgr.14156) 7 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:41:16.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:15.109555+0000 mon.smithi158 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:16.368 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:41:15.109865+0000 mgr.smithi158.gmoobf (mgr.14156) 8 : cephadm [INF] Added host smithi192 2024-02-20T13:41:17.386 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- 172.21.15.158:0/1450788331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 msgr2=0x7fa8280f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 --2- 172.21.15.158:0/1450788331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8280f58a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa824009fb0 tx=0x7fa824009b30).stop 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- 172.21.15.158:0/1450788331 shutdown_connections 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 --2- 172.21.15.158:0/1450788331 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8280f58a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- 172.21.15.158:0/1450788331 >> 172.21.15.158:0/1450788331 conn(0x7fa8280f0be0 msgr2=0x7fa8280f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- 172.21.15.158:0/1450788331 shutdown_connections 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- 172.21.15.158:0/1450788331 wait complete. 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 Processor -- start 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- start start 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82fda8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828060320 con 0x7fa8280f5480 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82db44700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82db44700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:37914/0 (socket says 172.21.15.158:37914) 2024-02-20T13:41:17.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.377+0000 7fa82db44700 1 -- 172.21.15.158:0/415119862 learned_addr learned my addr 172.21.15.158:0/415119862 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa82db44700 1 -- 172.21.15.158:0/415119862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824005040 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa82db44700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa8240089f0 tx=0x7fa82400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa824009320 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa824013ec0 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa824014d30 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa828060520 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.378+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa828109ba0 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa8240095f0 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa81e7fc700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 0x7fa814033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa824042390 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa82d343700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 0x7fa814033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828045bb0 con 0x7fa8280f5480 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.379+0000 7fa82d343700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 0x7fa814033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa8180096a0 tx=0x7fa818006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:17.388 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.383+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa82401d7f0 con 0x7fa8280f5480 2024-02-20T13:41:17.528 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.538+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8280f1de0 con 0x7fa814031090 2024-02-20T13:41:17.529 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.540+0000 7fa81e7fc700 1 -- 172.21.15.158:0/415119862 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fa8280f1de0 con 0x7fa814031090 2024-02-20T13:41:17.530 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-20T13:41:17.530 INFO:teuthology.orchestra.run.smithi158.stdout:[{"addr": "172.21.15.158", "hostname": "smithi158", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-02-20T13:41:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 msgr2=0x7fa814033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 0x7fa814033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa8180096a0 tx=0x7fa818006b40).stop 2024-02-20T13:41:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 msgr2=0x7fa8281094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa8240089f0 tx=0x7fa82400b040).stop 2024-02-20T13:41:17.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 shutdown_connections 2024-02-20T13:41:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa814031090 0x7fa814033550 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 --2- 172.21.15.158:0/415119862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8280f5480 0x7fa8281094f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 >> 172.21.15.158:0/415119862 conn(0x7fa8280f0be0 msgr2=0x7fa8280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 shutdown_connections 2024-02-20T13:41:17.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:17.541+0000 7fa82fda8700 1 -- 172.21.15.158:0/415119862 wait complete. 2024-02-20T13:41:17.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:15.323919+0000 mgr.smithi158.gmoobf (mgr.14156) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:18.163 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T13:41:18.163 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T13:41:19.511 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:17.324215+0000 mgr.smithi158.gmoobf (mgr.14156) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:19.511 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:17.540371+0000 mgr.smithi158.gmoobf (mgr.14156) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T13:41:19.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- 172.21.15.158:0/2918446112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 msgr2=0x7f78a00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 --2- 172.21.15.158:0/2918446112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a00f58a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7894009fb0 tx=0x7f7894009b30).stop 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- 172.21.15.158:0/2918446112 shutdown_connections 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 --2- 172.21.15.158:0/2918446112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a00f58a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- 172.21.15.158:0/2918446112 >> 172.21.15.158:0/2918446112 conn(0x7f78a00f0be0 msgr2=0x7f78a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- 172.21.15.158:0/2918446112 shutdown_connections 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- 172.21.15.158:0/2918446112 wait complete. 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 Processor -- start 2024-02-20T13:41:19.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.968+0000 7f78a6c23700 1 -- start start 2024-02-20T13:41:19.961 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a6c23700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a6c23700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0105be0 con 0x7f78a00f5480 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a49bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a49bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:51400/0 (socket says 172.21.15.158:51400) 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a49bf700 1 -- 172.21.15.158:0/1025133044 learned_addr learned my addr 172.21.15.158:0/1025133044 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a49bf700 1 -- 172.21.15.158:0/1025133044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894005040 con 0x7f78a00f5480 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a49bf700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f78940089f0 tx=0x7f789400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7894009320 con 0x7f78a00f5480 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7894013ec0 con 0x7f78a00f5480 2024-02-20T13:41:19.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7894014d30 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a0105de0 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.969+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a0106270 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.970+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f78940263a0 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.970+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0045bb0 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.971+0000 7f789d7fa700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 0x7f7890033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.971+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f78940423c0 con 0x7f78a00f5480 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.971+0000 7f789ffff700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 0x7f7890033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:19.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.971+0000 7f789ffff700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 0x7f7890033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f788c0096a0 tx=0x7f788c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:19.964 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:19.975+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7894026bf0 con 0x7f78a00f5480 2024-02-20T13:41:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:20.132+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f78a00faf70 con 0x7f78a00f5480 2024-02-20T13:41:21.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.116+0000 7f789d7fa700 1 -- 172.21.15.158:0/1025133044 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f789401db70 con 0x7f78a00f5480 2024-02-20T13:41:21.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 msgr2=0x7f7890033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:21.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 0x7f7890033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f788c0096a0 tx=0x7f788c006b40).stop 2024-02-20T13:41:21.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 msgr2=0x7f78a01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:21.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f78940089f0 tx=0x7f789400b040).stop 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 shutdown_connections 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7890031090 0x7f7890033550 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 --2- 172.21.15.158:0/1025133044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78a00f5480 0x7f78a01056f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 >> 172.21.15.158:0/1025133044 conn(0x7f78a00f0be0 msgr2=0x7f78a00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 shutdown_connections 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-20T13:41:21.118+0000 7f78a6c23700 1 -- 172.21.15.158:0/1025133044 wait complete. 2024-02-20T13:41:21.109 INFO:teuthology.orchestra.run.smithi158.stderr:adjusted tunables profile to default 2024-02-20T13:41:21.461 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:19.324534+0000 mgr.smithi158.gmoobf (mgr.14156) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:21.461 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:20.133179+0000 mon.smithi158 (mon.0) 102 : audit [INF] from='client.? 172.21.15.158:0/1025133044' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T13:41:21.757 INFO:tasks.cephadm:Adding mon.smithi158 on smithi158 2024-02-20T13:41:21.757 INFO:tasks.cephadm:Adding mon.smithi192 on smithi192 2024-02-20T13:41:21.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi158:172.21.15.158=smithi158;smithi192:172.21.15.192=smithi192' 2024-02-20T13:41:22.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:21.116620+0000 mon.smithi158 (mon.0) 103 : audit [INF] from='client.? 172.21.15.158:0/1025133044' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T13:41:22.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:21.116684+0000 mon.smithi158 (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-20T13:41:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:21.324836+0000 mgr.smithi158.gmoobf (mgr.14156) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:23.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 -- 172.21.15.192:0/1450474503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 msgr2=0x7fdda80f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:23.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 --2- 172.21.15.192:0/1450474503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda80f9160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdd9c009fb0 tx=0x7fdd9c009b30).stop 2024-02-20T13:41:23.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 -- 172.21.15.192:0/1450474503 shutdown_connections 2024-02-20T13:41:23.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 --2- 172.21.15.192:0/1450474503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda80f9160 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:23.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 -- 172.21.15.192:0/1450474503 >> 172.21.15.192:0/1450474503 conn(0x7fdda80f0be0 msgr2=0x7fdda80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:23.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 -- 172.21.15.192:0/1450474503 shutdown_connections 2024-02-20T13:41:23.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.599+0000 7fddae874700 1 -- 172.21.15.192:0/1450474503 wait complete. 2024-02-20T13:41:23.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fddae874700 1 Processor -- start 2024-02-20T13:41:23.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fddae874700 1 -- start start 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fddae874700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fddae874700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8106c40 con 0x7fdda8060360 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fdda7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fdda7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:47886/0 (socket says 172.21.15.192:47886) 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.600+0000 7fdda7fff700 1 -- 172.21.15.192:0/3572951183 learned_addr learned my addr 172.21.15.192:0/3572951183 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fdda7fff700 1 -- 172.21.15.192:0/3572951183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c005040 con 0x7fdda8060360 2024-02-20T13:41:23.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fdda7fff700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdd9c000c00 tx=0x7fdd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c0095f0 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda8106e40 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda81072d0 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdd9c007e50 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.601+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c014e00 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.602+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdd9c025070 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.602+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda8045bb0 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.602+0000 7fdda4ff9700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 0x7fdd900335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.602+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdd9c041e40 con 0x7fdda8060360 2024-02-20T13:41:23.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.602+0000 7fdda77fe700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 0x7fdd900335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:23.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.603+0000 7fdda77fe700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 0x7fdd900335a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdd980096a0 tx=0x7fdd98006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:23.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.607+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdd9c00f300 con 0x7fdda8060360 2024-02-20T13:41:23.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.764+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 --> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi158:172.21.15.158=smithi158;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7fdda80f1d70 con 0x7fdd900310e0 2024-02-20T13:41:23.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.770+0000 7fdda4ff9700 1 -- 172.21.15.192:0/3572951183 <== mgr.14156 v2:172.21.15.158:6800/770403169 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdda80f1d70 con 0x7fdd900310e0 2024-02-20T13:41:23.760 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-02-20T13:41:23.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 msgr2=0x7fdd900335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 0x7fdd900335a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdd980096a0 tx=0x7fdd98006b40).stop 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 msgr2=0x7fdda8106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdd9c000c00 tx=0x7fdd9c00b040).stop 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 shutdown_connections 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd900310e0 0x7fdd900335a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 --2- 172.21.15.192:0/3572951183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdda8060360 0x7fdda8106750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 >> 172.21.15.192:0/3572951183 conn(0x7fdda80f0be0 msgr2=0x7fdda80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 shutdown_connections 2024-02-20T13:41:23.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:23.771+0000 7fddae874700 1 -- 172.21.15.192:0/3572951183 wait complete. 2024-02-20T13:41:24.401 DEBUG:teuthology.orchestra.run.smithi192:mon.smithi192> sudo journalctl -f -n 0 -u ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192.service 2024-02-20T13:41:24.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:24.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:24.432 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:-- Logs begin at Tue 2024-02-20 13:28:13 UTC. -- 2024-02-20T13:41:25.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:23.325151+0000 mgr.smithi158.gmoobf (mgr.14156) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:25.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:23.766304+0000 mgr.smithi158.gmoobf (mgr.14156) 15 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi158:172.21.15.158=smithi158;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:41:25.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:41:23.767703+0000 mgr.smithi158.gmoobf (mgr.14156) 16 : cephadm [INF] Saving service mon spec with placement smithi158:172.21.15.158=smithi158;smithi192:172.21.15.192=smithi192;count:2 2024-02-20T13:41:25.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:23.770284+0000 mon.smithi158 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:41:26.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.316+0000 7f3c2ab99700 1 -- 172.21.15.192:0/88850608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 msgr2=0x7f3c240f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.316+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/88850608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c240f58d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3c18009fb0 tx=0x7f3c18009b30).stop 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.316+0000 7f3c2ab99700 1 -- 172.21.15.192:0/88850608 shutdown_connections 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.316+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/88850608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c240f58d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.316+0000 7f3c2ab99700 1 -- 172.21.15.192:0/88850608 >> 172.21.15.192:0/88850608 conn(0x7f3c240f0c10 msgr2=0x7f3c240f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 -- 172.21.15.192:0/88850608 shutdown_connections 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 -- 172.21.15.192:0/88850608 wait complete. 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 Processor -- start 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 -- start start 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.317+0000 7f3c2ab99700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c24060320 con 0x7f3c240f54b0 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c28935700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c28935700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:47910/0 (socket says 172.21.15.192:47910) 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c28935700 1 -- 172.21.15.192:0/3664874313 learned_addr learned my addr 172.21.15.192:0/3664874313 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c28935700 1 -- 172.21.15.192:0/3664874313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18005040 con 0x7f3c240f54b0 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c28935700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3c180089f0 tx=0x7f3c1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:26.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c18009320 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c24060520 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3c18013ec0 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c18014d30 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.318+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c24109bd0 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.319+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3c18014420 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.319+0000 7f3c217fa700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 0x7f3c14033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.319+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3c18042390 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.319+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c24045bb0 con 0x7f3c240f54b0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.320+0000 7f3c23fff700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 0x7f3c14033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:26.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.320+0000 7f3c23fff700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 0x7f3c14033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3c100096a0 tx=0x7f3c10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:26.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.324+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3c18014750 con 0x7f3c240f54b0 2024-02-20T13:41:26.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.531+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c2410a280 con 0x7f3c240f54b0 2024-02-20T13:41:26.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.532+0000 7f3c217fa700 1 -- 172.21.15.192:0/3664874313 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3c1801d7f0 con 0x7f3c240f54b0 2024-02-20T13:41:26.523 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:26.523 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:26.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 msgr2=0x7f3c14033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:26.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 0x7f3c14033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3c100096a0 tx=0x7f3c10006b40).stop 2024-02-20T13:41:26.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 msgr2=0x7f3c24109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:26.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3c180089f0 tx=0x7f3c1800b040).stop 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 shutdown_connections 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c14031090 0x7f3c14033550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 --2- 172.21.15.192:0/3664874313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c240f54b0 0x7f3c24109520 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 >> 172.21.15.192:0/3664874313 conn(0x7f3c240f0c10 msgr2=0x7f3c240f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 shutdown_connections 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:26.534+0000 7f3c2ab99700 1 -- 172.21.15.192:0/3664874313 wait complete. 2024-02-20T13:41:26.525 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:27.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:25.325401+0000 mgr.smithi158.gmoobf (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:27.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:26.533505+0000 mon.smithi158 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.192:0/3664874313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:28.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:28.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:29.036 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:27.325746+0000 mgr.smithi158.gmoobf (mgr.14156) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:29.973 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 -- 172.21.15.192:0/1743900510 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 msgr2=0x7f45440f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:29.973 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 --2- 172.21.15.192:0/1743900510 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440f58a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4534009fb0 tx=0x7f4534009b30).stop 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 -- 172.21.15.192:0/1743900510 shutdown_connections 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 --2- 172.21.15.192:0/1743900510 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440f58a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 -- 172.21.15.192:0/1743900510 >> 172.21.15.192:0/1743900510 conn(0x7f45440f0be0 msgr2=0x7f45440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 -- 172.21.15.192:0/1743900510 shutdown_connections 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.981+0000 7f454c6b5700 1 -- 172.21.15.192:0/1743900510 wait complete. 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454c6b5700 1 Processor -- start 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454c6b5700 1 -- start start 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454c6b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454c6b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440fb8e0 con 0x7f45440f5480 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454a451700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454a451700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:47928/0 (socket says 172.21.15.192:47928) 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454a451700 1 -- 172.21.15.192:0/2627278999 learned_addr learned my addr 172.21.15.192:0/2627278999 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454a451700 1 -- 172.21.15.192:0/2627278999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534005040 con 0x7f45440f5480 2024-02-20T13:41:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.982+0000 7f454a451700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f45340089f0 tx=0x7f453400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4534009320 con 0x7f45440f5480 2024-02-20T13:41:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4534013ec0 con 0x7f45440f5480 2024-02-20T13:41:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45440fba80 con 0x7f45440f5480 2024-02-20T13:41:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4534014d30 con 0x7f45440f5480 2024-02-20T13:41:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45440fbf10 con 0x7f45440f5480 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.983+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f45340263a0 con 0x7f45440f5480 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.984+0000 7f453affd700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 0x7f4530033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.984+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45440591e0 con 0x7f45440f5480 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.984+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4534042440 con 0x7f45440f5480 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.984+0000 7f4549c50700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 0x7f4530033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.984+0000 7f4549c50700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 0x7f4530033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f45400096a0 tx=0x7f4540006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:29.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:29.989+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4534012510 con 0x7f45440f5480 2024-02-20T13:41:30.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.196+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45440fc6b0 con 0x7f45440f5480 2024-02-20T13:41:30.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.197+0000 7f453affd700 1 -- 172.21.15.192:0/2627278999 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4534012330 con 0x7f45440f5480 2024-02-20T13:41:30.187 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:30.187 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:30.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 msgr2=0x7f4530033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 0x7f4530033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f45400096a0 tx=0x7f4540006b40).stop 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 msgr2=0x7f45440fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f45340089f0 tx=0x7f453400b040).stop 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 shutdown_connections 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4530031090 0x7f4530033550 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 --2- 172.21.15.192:0/2627278999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45440f5480 0x7f45440fd460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 >> 172.21.15.192:0/2627278999 conn(0x7f45440f0be0 msgr2=0x7f45440f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 shutdown_connections 2024-02-20T13:41:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:30.198+0000 7f454c6b5700 1 -- 172.21.15.192:0/2627278999 wait complete. 2024-02-20T13:41:30.190 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:31.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:29.326036+0000 mgr.smithi158.gmoobf (mgr.14156) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:31.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:30.198090+0000 mon.smithi158 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.192:0/2627278999' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:31.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:31.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:33.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:31.326374+0000 mgr.smithi158.gmoobf (mgr.14156) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 -- 172.21.15.192:0/4226119523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 msgr2=0x7f5afc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 --2- 172.21.15.192:0/4226119523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc0f69b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5ae4004760 tx=0x7f5ae4009ad0).stop 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 -- 172.21.15.192:0/4226119523 shutdown_connections 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 --2- 172.21.15.192:0/4226119523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc0f69b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 -- 172.21.15.192:0/4226119523 >> 172.21.15.192:0/4226119523 conn(0x7f5afc0f1cd0 msgr2=0x7f5afc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 -- 172.21.15.192:0/4226119523 shutdown_connections 2024-02-20T13:41:33.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.651+0000 7f5b00c25700 1 -- 172.21.15.192:0/4226119523 wait complete. 2024-02-20T13:41:33.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5b00c25700 1 Processor -- start 2024-02-20T13:41:33.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5b00c25700 1 -- start start 2024-02-20T13:41:33.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5b00c25700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:33.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5b00c25700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc106cb0 con 0x7f5afc0f6590 2024-02-20T13:41:33.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5afa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5afa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37788/0 (socket says 172.21.15.192:37788) 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5afa59c700 1 -- 172.21.15.192:0/3268250825 learned_addr learned my addr 172.21.15.192:0/3268250825 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5afa59c700 1 -- 172.21.15.192:0/3268250825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009d70 con 0x7f5afc0f6590 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.652+0000 7f5afa59c700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5ae4013f10 tx=0x7f5ae4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.653+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae400e040 con 0x7f5afc0f6590 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.653+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc106eb0 con 0x7f5afc0f6590 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.653+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5ae4007df0 con 0x7f5afc0f6590 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.653+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae4014d60 con 0x7f5afc0f6590 2024-02-20T13:41:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.653+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc107340 con 0x7f5afc0f6590 2024-02-20T13:41:33.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc045bb0 con 0x7f5afc0f6590 2024-02-20T13:41:33.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f5ae4014520 con 0x7f5afc0f6590 2024-02-20T13:41:33.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5af2ffd700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 0x7f5ae8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:33.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5ae4042460 con 0x7f5afc0f6590 2024-02-20T13:41:33.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5af9d9b700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 0x7f5ae8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:33.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.654+0000 7f5af9d9b700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 0x7f5ae8033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5aec0096a0 tx=0x7f5aec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:33.648 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.658+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5ae4007960 con 0x7f5afc0f6590 2024-02-20T13:41:33.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.865+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5afc1081f0 con 0x7f5afc0f6590 2024-02-20T13:41:33.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.865+0000 7f5af2ffd700 1 -- 172.21.15.192:0/3268250825 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5ae4026760 con 0x7f5afc0f6590 2024-02-20T13:41:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:33.856 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:33.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 msgr2=0x7f5ae8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:33.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 0x7f5ae8033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5aec0096a0 tx=0x7f5aec006b40).stop 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 msgr2=0x7f5afc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5ae4013f10 tx=0x7f5ae4009ad0).stop 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 shutdown_connections 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5ae8031090 0x7f5ae8033550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 --2- 172.21.15.192:0/3268250825 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5afc0f6590 0x7f5afc1067c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 >> 172.21.15.192:0/3268250825 conn(0x7f5afc0f1cd0 msgr2=0x7f5afc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 shutdown_connections 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:33.867+0000 7f5b00c25700 1 -- 172.21.15.192:0/3268250825 wait complete. 2024-02-20T13:41:33.858 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:34.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:33.866841+0000 mon.smithi158 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.192:0/3268250825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:35.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:35.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:35.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:33.326696+0000 mgr.smithi158.gmoobf (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:37.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 -- 172.21.15.192:0/2839931177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 msgr2=0x7fb3100f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 --2- 172.21.15.192:0/2839931177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb3100f58d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb304009fb0 tx=0x7fb304009b30).stop 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 -- 172.21.15.192:0/2839931177 shutdown_connections 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 --2- 172.21.15.192:0/2839931177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb3100f58d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 -- 172.21.15.192:0/2839931177 >> 172.21.15.192:0/2839931177 conn(0x7fb3100f0c10 msgr2=0x7fb3100f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 -- 172.21.15.192:0/2839931177 shutdown_connections 2024-02-20T13:41:37.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.245+0000 7fb3150f2700 1 -- 172.21.15.192:0/2839931177 wait complete. 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb3150f2700 1 Processor -- start 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb3150f2700 1 -- start start 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb3150f2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb3150f2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb310060300 con 0x7fb3100f54b0 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb30ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb30ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37806/0 (socket says 172.21.15.192:37806) 2024-02-20T13:41:37.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb30ed9d700 1 -- 172.21.15.192:0/4217577065 learned_addr learned my addr 172.21.15.192:0/4217577065 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb30ed9d700 1 -- 172.21.15.192:0/4217577065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304005040 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.246+0000 7fb30ed9d700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb3040089f0 tx=0x7fb30400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304009320 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb310060500 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb304013ec0 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304014d30 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb310109bf0 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.247+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fb304014420 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.248+0000 7fb2f77fe700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 0x7fb2f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.248+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb304042390 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.248+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3100591e0 con 0x7fb3100f54b0 2024-02-20T13:41:37.238 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.248+0000 7fb30e59c700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 0x7fb2f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:37.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.248+0000 7fb30e59c700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 0x7fb2f8033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb3000096a0 tx=0x7fb300006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:37.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.253+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb304014750 con 0x7fb3100f54b0 2024-02-20T13:41:37.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.458+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb310045bb0 con 0x7fb3100f54b0 2024-02-20T13:41:37.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.458+0000 7fb2f77fe700 1 -- 172.21.15.192:0/4217577065 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb30401d7f0 con 0x7fb3100f54b0 2024-02-20T13:41:37.448 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:37.448 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:37.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 msgr2=0x7fb2f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:37.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 0x7fb2f8033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb3000096a0 tx=0x7fb300006b40).stop 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 msgr2=0x7fb310109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb3040089f0 tx=0x7fb30400b040).stop 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 shutdown_connections 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb2f8031090 0x7fb2f8033550 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 --2- 172.21.15.192:0/4217577065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3100f54b0 0x7fb310109540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 >> 172.21.15.192:0/4217577065 conn(0x7fb3100f0c10 msgr2=0x7fb3100f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 shutdown_connections 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:37.460+0000 7fb3150f2700 1 -- 172.21.15.192:0/4217577065 wait complete. 2024-02-20T13:41:37.450 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:37.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:35.326898+0000 mgr.smithi158.gmoobf (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:38.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:37.458824+0000 mon.smithi158 (mon.0) 109 : audit [DBG] from='client.? 172.21.15.192:0/4217577065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:39.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:39.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:39.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:37.327234+0000 mgr.smithi158.gmoobf (mgr.14156) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:40.785 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 -- 172.21.15.192:0/1683179447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 msgr2=0x7f3ebc0f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/1683179447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc0f91c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3eac009fb0 tx=0x7f3eac009b30).stop 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 -- 172.21.15.192:0/1683179447 shutdown_connections 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/1683179447 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc0f91c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 -- 172.21.15.192:0/1683179447 >> 172.21.15.192:0/1683179447 conn(0x7f3ebc0f0be0 msgr2=0x7f3ebc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 -- 172.21.15.192:0/1683179447 shutdown_connections 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.791+0000 7f3ec0a93700 1 -- 172.21.15.192:0/1683179447 wait complete. 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3ec0a93700 1 Processor -- start 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3ec0a93700 1 -- start start 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3ec0a93700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3ec0a93700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc106c70 con 0x7f3ebc060300 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3eba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3eba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37820/0 (socket says 172.21.15.192:37820) 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.792+0000 7f3eba59c700 1 -- 172.21.15.192:0/3975598281 learned_addr learned my addr 172.21.15.192:0/3975598281 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3eba59c700 1 -- 172.21.15.192:0/3975598281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac005040 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3eba59c700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3eac009f80 tx=0x7f3eac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eac009320 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc106e70 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3eac013680 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eac01eda0 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.793+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ebc107300 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.794+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3eac01e420 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.794+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ebc045bb0 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.794+0000 7f3eb2ffd700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 0x7f3ea8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.794+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3eac042360 con 0x7f3ebc060300 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.794+0000 7f3eb9d9b700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 0x7f3ea8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:40.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.795+0000 7f3eb9d9b700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 0x7f3ea8033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3ea40096a0 tx=0x7f3ea4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:40.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:40.799+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3eac01d380 con 0x7f3ebc060300 2024-02-20T13:41:40.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.004+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ebc1081b0 con 0x7f3ebc060300 2024-02-20T13:41:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.005+0000 7f3eb2ffd700 1 -- 172.21.15.192:0/3975598281 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3eac004490 con 0x7f3ebc060300 2024-02-20T13:41:40.994 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:40.994 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 msgr2=0x7f3ea8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 0x7f3ea8033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3ea40096a0 tx=0x7f3ea4006b40).stop 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 msgr2=0x7f3ebc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3eac009f80 tx=0x7f3eac00b040).stop 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 shutdown_connections 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ea8031090 0x7f3ea8033550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 --2- 172.21.15.192:0/3975598281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ebc060300 0x7f3ebc106780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 >> 172.21.15.192:0/3975598281 conn(0x7f3ebc0f0be0 msgr2=0x7f3ebc0f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 shutdown_connections 2024-02-20T13:41:40.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:41.006+0000 7f3ec0a93700 1 -- 172.21.15.192:0/3975598281 wait complete. 2024-02-20T13:41:40.997 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:41.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:39.327619+0000 mgr.smithi158.gmoobf (mgr.14156) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:41.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:41.004980+0000 mon.smithi158 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.192:0/3975598281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:42.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:42.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:43.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:41.327873+0000 mgr.smithi158.gmoobf (mgr.14156) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:44.264 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.274+0000 7f0b90268700 1 -- 172.21.15.192:0/3348334530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 msgr2=0x7f0b880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.274+0000 7f0b90268700 1 --2- 172.21.15.192:0/3348334530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b880f69b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0b78004760 tx=0x7f0b78009ad0).stop 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- 172.21.15.192:0/3348334530 shutdown_connections 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 --2- 172.21.15.192:0/3348334530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b880f69b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- 172.21.15.192:0/3348334530 >> 172.21.15.192:0/3348334530 conn(0x7f0b880f1cd0 msgr2=0x7f0b880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- 172.21.15.192:0/3348334530 shutdown_connections 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- 172.21.15.192:0/3348334530 wait complete. 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 Processor -- start 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- start start 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.275+0000 7f0b90268700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88106cb0 con 0x7f0b880f6590 2024-02-20T13:41:44.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b8e004700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:44.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b8e004700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36858/0 (socket says 172.21.15.192:36858) 2024-02-20T13:41:44.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b8e004700 1 -- 172.21.15.192:0/671223025 learned_addr learned my addr 172.21.15.192:0/671223025 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:44.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b8e004700 1 -- 172.21.15.192:0/671223025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009d70 con 0x7f0b880f6590 2024-02-20T13:41:44.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b8e004700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0b78013f10 tx=0x7f0b78009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:44.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b7800e040 con 0x7f0b880f6590 2024-02-20T13:41:44.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88106eb0 con 0x7f0b880f6590 2024-02-20T13:41:44.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0b78007df0 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b78014d60 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.276+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88107340 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.277+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f0b7801f460 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.277+0000 7f0b7effd700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 0x7f0b74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.277+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88045bb0 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.277+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0b78042470 con 0x7f0b880f6590 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.278+0000 7f0b8d803700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 0x7f0b74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.278+0000 7f0b8d803700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 0x7f0b74033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0b840096a0 tx=0x7f0b84006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:44.272 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.282+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0b780283f0 con 0x7f0b880f6590 2024-02-20T13:41:44.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.491+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b881081c0 con 0x7f0b880f6590 2024-02-20T13:41:44.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.492+0000 7f0b7effd700 1 -- 172.21.15.192:0/671223025 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0b78012330 con 0x7f0b880f6590 2024-02-20T13:41:44.481 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:44.481 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.493+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 msgr2=0x7f0b74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.493+0000 7f0b90268700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 0x7f0b74033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0b840096a0 tx=0x7f0b84006b40).stop 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 msgr2=0x7f0b881067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0b78013f10 tx=0x7f0b78009ad0).stop 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 shutdown_connections 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0b74031090 0x7f0b74033550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 --2- 172.21.15.192:0/671223025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b880f6590 0x7f0b881067c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 >> 172.21.15.192:0/671223025 conn(0x7f0b880f1cd0 msgr2=0x7f0b880f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:44.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 shutdown_connections 2024-02-20T13:41:44.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:44.494+0000 7f0b90268700 1 -- 172.21.15.192:0/671223025 wait complete. 2024-02-20T13:41:44.484 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:45.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:43.328197+0000 mgr.smithi158.gmoobf (mgr.14156) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:45.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:44.492500+0000 mon.smithi158 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.192:0/671223025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:46.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:46.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:45.328501+0000 mgr.smithi158.gmoobf (mgr.14156) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 -- 172.21.15.192:0/2892255516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 msgr2=0x7fcb580f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/2892255516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb580f6800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcb4c009fb0 tx=0x7fcb4c009b30).stop 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 -- 172.21.15.192:0/2892255516 shutdown_connections 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/2892255516 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb580f6800 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 -- 172.21.15.192:0/2892255516 >> 172.21.15.192:0/2892255516 conn(0x7fcb580f1b40 msgr2=0x7fcb580f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 -- 172.21.15.192:0/2892255516 shutdown_connections 2024-02-20T13:41:47.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.823+0000 7fcb5c94b700 1 -- 172.21.15.192:0/2892255516 wait complete. 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.824+0000 7fcb5c94b700 1 Processor -- start 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.824+0000 7fcb5c94b700 1 -- start start 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.824+0000 7fcb5c94b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb5c94b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58106b10 con 0x7fcb580f63e0 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb56d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb56d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36882/0 (socket says 172.21.15.192:36882) 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb56d9d700 1 -- 172.21.15.192:0/3706306675 learned_addr learned my addr 172.21.15.192:0/3706306675 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb56d9d700 1 -- 172.21.15.192:0/3706306675 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c005040 con 0x7fcb580f63e0 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb56d9d700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcb4c0089f0 tx=0x7fcb4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c009320 con 0x7fcb580f63e0 2024-02-20T13:41:47.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb58106d10 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcb4c013ec0 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.825+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c014d30 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.826+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581071a0 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.826+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fcb4c0095f0 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.826+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb58045bb0 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.826+0000 7fcb3f7fe700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 0x7fcb40033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.827+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcb4c042480 con 0x7fcb580f63e0 2024-02-20T13:41:47.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.827+0000 7fcb5659c700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 0x7fcb40033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:47.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.827+0000 7fcb5659c700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 0x7fcb40033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:47.820 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:47.831+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb4c01d380 con 0x7fcb580f63e0 2024-02-20T13:41:48.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.036+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb58108050 con 0x7fcb580f63e0 2024-02-20T13:41:48.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.037+0000 7fcb3f7fe700 1 -- 172.21.15.192:0/3706306675 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb4c01d380 con 0x7fcb580f63e0 2024-02-20T13:41:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:48.027 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 msgr2=0x7fcb40033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 0x7fcb40033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).stop 2024-02-20T13:41:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 msgr2=0x7fcb58106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcb4c0089f0 tx=0x7fcb4c00b040).stop 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 shutdown_connections 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcb40031090 0x7fcb40033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 --2- 172.21.15.192:0/3706306675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb580f63e0 0x7fcb58106620 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 >> 172.21.15.192:0/3706306675 conn(0x7fcb580f1b40 msgr2=0x7fcb580f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 shutdown_connections 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:48.039+0000 7fcb5c94b700 1 -- 172.21.15.192:0/3706306675 wait complete. 2024-02-20T13:41:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:48.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:48.037455+0000 mon.smithi158 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.192:0/3706306675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:49.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:47.328863+0000 mgr.smithi158.gmoobf (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:49.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:49.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:51.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.485+0000 7f7173fff700 1 -- 172.21.15.192:0/1050412923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 msgr2=0x7f71740f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:51.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.485+0000 7f7173fff700 1 --2- 172.21.15.192:0/1050412923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71740f8fb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7164009fb0 tx=0x7f7164009b30).stop 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.485+0000 7f7173fff700 1 -- 172.21.15.192:0/1050412923 shutdown_connections 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.485+0000 7f7173fff700 1 --2- 172.21.15.192:0/1050412923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71740f8fb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.485+0000 7f7173fff700 1 -- 172.21.15.192:0/1050412923 >> 172.21.15.192:0/1050412923 conn(0x7f71740f0a30 msgr2=0x7f71740f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 -- 172.21.15.192:0/1050412923 shutdown_connections 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 -- 172.21.15.192:0/1050412923 wait complete. 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 Processor -- start 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 -- start start 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.486+0000 7f7173fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7174106aa0 con 0x7f7174060360 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7172ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7172ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36896/0 (socket says 172.21.15.192:36896) 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7172ffd700 1 -- 172.21.15.192:0/1261882212 learned_addr learned my addr 172.21.15.192:0/1261882212 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7172ffd700 1 -- 172.21.15.192:0/1261882212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7164005040 con 0x7f7174060360 2024-02-20T13:41:51.478 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7172ffd700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7164009f80 tx=0x7f716400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:51.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7164009320 con 0x7f7174060360 2024-02-20T13:41:51.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7174106ca0 con 0x7f7174060360 2024-02-20T13:41:51.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7164013680 con 0x7f7174060360 2024-02-20T13:41:51.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f716401eda0 con 0x7f7174060360 2024-02-20T13:41:51.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.487+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7174107130 con 0x7f7174060360 2024-02-20T13:41:51.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.488+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f7164014420 con 0x7f7174060360 2024-02-20T13:41:51.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.488+0000 7f715b7fe700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 0x7f715c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:51.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.488+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7164042440 con 0x7f7174060360 2024-02-20T13:41:51.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.488+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71740591e0 con 0x7f7174060360 2024-02-20T13:41:51.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.488+0000 7f71727fc700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 0x7f715c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:51.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.489+0000 7f71727fc700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 0x7f715c033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f71680096a0 tx=0x7f7168006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:51.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.493+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7164014700 con 0x7f7174060360 2024-02-20T13:41:51.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:49.329220+0000 mgr.smithi158.gmoobf (mgr.14156) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:51.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.701+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7174045bb0 con 0x7f7174060360 2024-02-20T13:41:51.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.701+0000 7f715b7fe700 1 -- 172.21.15.192:0/1261882212 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f716401c320 con 0x7f7174060360 2024-02-20T13:41:51.692 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:51.692 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 msgr2=0x7f715c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 0x7f715c033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f71680096a0 tx=0x7f7168006b40).stop 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 msgr2=0x7f71741065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7164009f80 tx=0x7f716400b040).stop 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 shutdown_connections 2024-02-20T13:41:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f715c031090 0x7f715c033550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:51.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 --2- 172.21.15.192:0/1261882212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7174060360 0x7f71741065b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:51.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 >> 172.21.15.192:0/1261882212 conn(0x7f71740f0a30 msgr2=0x7f71740f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:51.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 shutdown_connections 2024-02-20T13:41:51.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:51.703+0000 7f7173fff700 1 -- 172.21.15.192:0/1261882212 wait complete. 2024-02-20T13:41:51.694 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:52.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:52 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:51.702885+0000 mon.smithi158 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.192:0/1261882212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:53.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:53.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:53.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:51.329526+0000 mgr.smithi158.gmoobf (mgr.14156) 30 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:55.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 -- 172.21.15.192:0/1241516763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 msgr2=0x7f653c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:55.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 --2- 172.21.15.192:0/1241516763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c0f58a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f652c009fb0 tx=0x7f652c009b30).stop 2024-02-20T13:41:55.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 -- 172.21.15.192:0/1241516763 shutdown_connections 2024-02-20T13:41:55.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 --2- 172.21.15.192:0/1241516763 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c0f58a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 -- 172.21.15.192:0/1241516763 >> 172.21.15.192:0/1241516763 conn(0x7f653c0f0be0 msgr2=0x7f653c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 -- 172.21.15.192:0/1241516763 shutdown_connections 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 -- 172.21.15.192:0/1241516763 wait complete. 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.191+0000 7f6541e15700 1 Processor -- start 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f6541e15700 1 -- start start 2024-02-20T13:41:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f6541e15700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f6541e15700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c105be0 con 0x7f653c0f5480 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f653b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f653b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48088/0 (socket says 172.21.15.192:48088) 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f653b7fe700 1 -- 172.21.15.192:0/4164235146 learned_addr learned my addr 172.21.15.192:0/4164235146 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f653b7fe700 1 -- 172.21.15.192:0/4164235146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c005040 con 0x7f653c0f5480 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f653b7fe700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f652c0089f0 tx=0x7f652c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.192+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f652c009320 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.193+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c105de0 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.193+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f652c013ec0 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.193+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f652c014d30 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.193+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c106270 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.193+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f652c0095f0 con 0x7f653c0f5480 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.194+0000 7f6523fff700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 0x7f65240335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:55.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.194+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c045bb0 con 0x7f653c0f5480 2024-02-20T13:41:55.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.194+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f652c042450 con 0x7f653c0f5480 2024-02-20T13:41:55.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.194+0000 7f653affd700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 0x7f65240335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:55.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.194+0000 7f653affd700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 0x7f65240335a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f65300096a0 tx=0x7f6530006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:55.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.198+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f652c01d7f0 con 0x7f653c0f5480 2024-02-20T13:41:55.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.403+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f653c0fb060 con 0x7f653c0f5480 2024-02-20T13:41:55.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.404+0000 7f6523fff700 1 -- 172.21.15.192:0/4164235146 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f652c01d7f0 con 0x7f653c0f5480 2024-02-20T13:41:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:55.394 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.405+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 msgr2=0x7f65240335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.405+0000 7f6541e15700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 0x7f65240335a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f65300096a0 tx=0x7f6530006b40).stop 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.405+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 msgr2=0x7f653c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.405+0000 7f6541e15700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f652c0089f0 tx=0x7f652c00b040).stop 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 shutdown_connections 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f65240310e0 0x7f65240335a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 --2- 172.21.15.192:0/4164235146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f653c0f5480 0x7f653c1056f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 >> 172.21.15.192:0/4164235146 conn(0x7f653c0f0be0 msgr2=0x7f653c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:55.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 shutdown_connections 2024-02-20T13:41:55.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:55.406+0000 7f6541e15700 1 -- 172.21.15.192:0/4164235146 wait complete. 2024-02-20T13:41:55.397 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:55.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:53.329851+0000 mgr.smithi158.gmoobf (mgr.14156) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:56.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:55.405547+0000 mon.smithi158 (mon.0) 114 : audit [DBG] from='client.? 172.21.15.192:0/4164235146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:41:56.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:55.458600+0000 mon.smithi158 (mon.0) 115 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:41:56.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:55.459642+0000 mon.smithi158 (mon.0) 116 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:41:56.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:41:56.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:41:57.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:55.330179+0000 mgr.smithi158.gmoobf (mgr.14156) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:58.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 -- 172.21.15.192:0/649304210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 msgr2=0x7f3cfc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:58.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 --2- 172.21.15.192:0/649304210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc0f56f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3cec009fb0 tx=0x7f3cec009b30).stop 2024-02-20T13:41:58.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 -- 172.21.15.192:0/649304210 shutdown_connections 2024-02-20T13:41:58.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 --2- 172.21.15.192:0/649304210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc0f56f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:58.769 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 -- 172.21.15.192:0/649304210 >> 172.21.15.192:0/649304210 conn(0x7f3cfc0f0a10 msgr2=0x7f3cfc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:58.769 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 -- 172.21.15.192:0/649304210 shutdown_connections 2024-02-20T13:41:58.769 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.777+0000 7f3d03462700 1 -- 172.21.15.192:0/649304210 wait complete. 2024-02-20T13:41:58.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.778+0000 7f3d03462700 1 Processor -- start 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.778+0000 7f3d03462700 1 -- start start 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.778+0000 7f3d03462700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d03462700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc105a30 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d02460700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d02460700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48104/0 (socket says 172.21.15.192:48104) 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d02460700 1 -- 172.21.15.192:0/46417664 learned_addr learned my addr 172.21.15.192:0/46417664 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d02460700 1 -- 172.21.15.192:0/46417664 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec005040 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3d02460700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3cec000c00 tx=0x7f3cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.779+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec0095f0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc105c30 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3cec007e50 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec014e70 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc1060c0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3cec0146b0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc045bb0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.780+0000 7f3cf2ffd700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 0x7f3ce8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.781+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3cec041d10 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.781+0000 7f3d01c5f700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 0x7f3ce8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:41:58.773 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.781+0000 7f3d01c5f700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 0x7f3ce8033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3cf80096a0 tx=0x7f3cf8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:41:58.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.785+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3cec01b4a0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.993+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cfc106f30 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.994+0000 7f3cf2ffd700 1 -- 172.21.15.192:0/46417664 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3cec01b2c0 con 0x7f3cfc0f52d0 2024-02-20T13:41:58.984 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:41:58.984 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 msgr2=0x7f3ce8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 0x7f3ce8033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3cf80096a0 tx=0x7f3cf8006b40).stop 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 msgr2=0x7f3cfc105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3cec000c00 tx=0x7f3cec00b040).stop 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 shutdown_connections 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ce8031090 0x7f3ce8033550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 --2- 172.21.15.192:0/46417664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cfc0f52d0 0x7f3cfc105540 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 >> 172.21.15.192:0/46417664 conn(0x7f3cfc0f0a10 msgr2=0x7f3cfc0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:41:58.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 shutdown_connections 2024-02-20T13:41:58.987 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:41:58.996+0000 7f3d03462700 1 -- 172.21.15.192:0/46417664 wait complete. 2024-02-20T13:41:58.987 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:41:59.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:57.330520+0000 mgr.smithi158.gmoobf (mgr.14156) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:41:59.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:41:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:41:58.995380+0000 mon.smithi158 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.192:0/46417664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:00.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:00.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:01.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:41:59.330863+0000 mgr.smithi158.gmoobf (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:02.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 -- 172.21.15.192:0/1499078282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 msgr2=0x7f60f00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:02.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 --2- 172.21.15.192:0/1499078282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f00f69b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f60e0004760 tx=0x7f60e0009ad0).stop 2024-02-20T13:42:02.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 -- 172.21.15.192:0/1499078282 shutdown_connections 2024-02-20T13:42:02.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 --2- 172.21.15.192:0/1499078282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f00f69b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:02.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 -- 172.21.15.192:0/1499078282 >> 172.21.15.192:0/1499078282 conn(0x7f60f00f1cd0 msgr2=0x7f60f00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:02.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 -- 172.21.15.192:0/1499078282 shutdown_connections 2024-02-20T13:42:02.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.449+0000 7f60f7410700 1 -- 172.21.15.192:0/1499078282 wait complete. 2024-02-20T13:42:02.447 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f7410700 1 Processor -- start 2024-02-20T13:42:02.448 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f7410700 1 -- start start 2024-02-20T13:42:02.448 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f7410700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:02.448 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f7410700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0106cb0 con 0x7f60f00f6590 2024-02-20T13:42:02.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f51ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:02.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f51ac700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:42692/0 (socket says 172.21.15.192:42692) 2024-02-20T13:42:02.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f51ac700 1 -- 172.21.15.192:0/3077244275 learned_addr learned my addr 172.21.15.192:0/3077244275 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:02.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f51ac700 1 -- 172.21.15.192:0/3077244275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0009d70 con 0x7f60f00f6590 2024-02-20T13:42:02.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.450+0000 7f60f51ac700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f60e0013f10 tx=0x7f60e0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:02.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.451+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60e000e040 con 0x7f60f00f6590 2024-02-20T13:42:02.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.451+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f60e0007df0 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.451+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f0106eb0 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.451+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60e0014d60 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.451+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f0107340 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f60e0014520 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60e5ffb700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 0x7f60dc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f0045bb0 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f60e0042400 con 0x7f60f00f6590 2024-02-20T13:42:02.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60f49ab700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 0x7f60dc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:02.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.452+0000 7f60f49ab700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 0x7f60dc0335a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f60ec0096a0 tx=0x7f60ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:02.453 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.456+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f60e0026b50 con 0x7f60f00f6590 2024-02-20T13:42:02.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.663+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60f01081f0 con 0x7f60f00f6590 2024-02-20T13:42:02.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.664+0000 7f60e5ffb700 1 -- 172.21.15.192:0/3077244275 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f60e0027350 con 0x7f60f00f6590 2024-02-20T13:42:02.654 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:02.654 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 msgr2=0x7f60dc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 0x7f60dc0335a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f60ec0096a0 tx=0x7f60ec006b40).stop 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 msgr2=0x7f60f01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f60e0013f10 tx=0x7f60e0009ad0).stop 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 shutdown_connections 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f60dc0310e0 0x7f60dc0335a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 --2- 172.21.15.192:0/3077244275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60f00f6590 0x7f60f01067c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 >> 172.21.15.192:0/3077244275 conn(0x7f60f00f1cd0 msgr2=0x7f60f00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 shutdown_connections 2024-02-20T13:42:02.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:02.665+0000 7f60f7410700 1 -- 172.21.15.192:0/3077244275 wait complete. 2024-02-20T13:42:02.657 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:03.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:01.331208+0000 mgr.smithi158.gmoobf (mgr.14156) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:03.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:02.665060+0000 mon.smithi158 (mon.0) 118 : audit [DBG] from='client.? 172.21.15.192:0/3077244275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:04.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:04.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:05.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:03.331581+0000 mgr.smithi158.gmoobf (mgr.14156) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:06.063 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 -- 172.21.15.192:0/2510530013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 msgr2=0x7fa9500f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:06.063 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 --2- 172.21.15.192:0/2510530013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9500f69b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fa940004760 tx=0x7fa940009ad0).stop 2024-02-20T13:42:06.063 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 -- 172.21.15.192:0/2510530013 shutdown_connections 2024-02-20T13:42:06.063 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 --2- 172.21.15.192:0/2510530013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9500f69b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 -- 172.21.15.192:0/2510530013 >> 172.21.15.192:0/2510530013 conn(0x7fa9500f1cd0 msgr2=0x7fa9500f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 -- 172.21.15.192:0/2510530013 shutdown_connections 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.072+0000 7fa958173700 1 -- 172.21.15.192:0/2510530013 wait complete. 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.073+0000 7fa958173700 1 Processor -- start 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.073+0000 7fa958173700 1 -- start start 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.073+0000 7fa958173700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.073+0000 7fa958173700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa950106cb0 con 0x7fa9500f6590 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.073+0000 7fa955f0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa955f0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:42724/0 (socket says 172.21.15.192:42724) 2024-02-20T13:42:06.064 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa955f0f700 1 -- 172.21.15.192:0/4011597323 learned_addr learned my addr 172.21.15.192:0/4011597323 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa955f0f700 1 -- 172.21.15.192:0/4011597323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa940009d70 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa955f0f700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa940004170 tx=0x7fa940009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa94000e040 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa940008ef0 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa940014c90 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa950106eb0 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.074+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa950107340 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.075+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa940014420 con 0x7fa9500f6590 2024-02-20T13:42:06.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.075+0000 7fa9467fc700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 0x7fa93c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:06.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.075+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa950045bb0 con 0x7fa9500f6590 2024-02-20T13:42:06.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.075+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa940042220 con 0x7fa9500f6590 2024-02-20T13:42:06.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.075+0000 7fa95570e700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 0x7fa93c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:06.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.076+0000 7fa95570e700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 0x7fa93c033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa94c0096a0 tx=0x7fa94c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:06.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.080+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa940026300 con 0x7fa9500f6590 2024-02-20T13:42:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.287+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa950108140 con 0x7fa9500f6590 2024-02-20T13:42:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.287+0000 7fa9467fc700 1 -- 172.21.15.192:0/4011597323 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa940012b20 con 0x7fa9500f6590 2024-02-20T13:42:06.278 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:06.278 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 msgr2=0x7fa93c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 0x7fa93c033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa94c0096a0 tx=0x7fa94c006b40).stop 2024-02-20T13:42:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 msgr2=0x7fa9501067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa940004170 tx=0x7fa940009ad0).stop 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 shutdown_connections 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa93c031090 0x7fa93c033550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 --2- 172.21.15.192:0/4011597323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9500f6590 0x7fa9501067c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 >> 172.21.15.192:0/4011597323 conn(0x7fa9500f1cd0 msgr2=0x7fa9500f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 shutdown_connections 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:06.289+0000 7fa958173700 1 -- 172.21.15.192:0/4011597323 wait complete. 2024-02-20T13:42:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:07.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:05.331926+0000 mgr.smithi158.gmoobf (mgr.14156) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:07.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:06.288681+0000 mon.smithi158 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.192:0/4011597323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:07.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:07.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:09.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:07.332183+0000 mgr.smithi158.gmoobf (mgr.14156) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:09.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/2606937540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 msgr2=0x7f2c280f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:09.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/2606937540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c280f67f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2c1c009fb0 tx=0x7f2c1c009b30).stop 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/2606937540 shutdown_connections 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/2606937540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c280f67f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/2606937540 >> 172.21.15.192:0/2606937540 conn(0x7f2c280f1b50 msgr2=0x7f2c280f3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/2606937540 shutdown_connections 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/2606937540 wait complete. 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 Processor -- start 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.740+0000 7f2c2d9fc700 1 -- start start 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2d9fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2d9fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28106b30 con 0x7f2c280f63d0 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2c9fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2c9fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:42740/0 (socket says 172.21.15.192:42740) 2024-02-20T13:42:09.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2c9fa700 1 -- 172.21.15.192:0/3636923937 learned_addr learned my addr 172.21.15.192:0/3636923937 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2c9fa700 1 -- 172.21.15.192:0/3636923937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c005040 con 0x7f2c280f63d0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2c9fa700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2c1c0089f0 tx=0x7f2c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c1c009320 con 0x7f2c280f63d0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2c1c013ec0 con 0x7f2c280f63d0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c28106d30 con 0x7f2c280f63d0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.741+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c1c014d30 con 0x7f2c280f63d0 2024-02-20T13:42:09.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.742+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c281071c0 con 0x7f2c280f63d0 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.742+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2c1c0095f0 con 0x7f2c280f63d0 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.742+0000 7f2c257fa700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 0x7f2c18033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.742+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c28045bb0 con 0x7f2c280f63d0 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.743+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2c1c042490 con 0x7f2c280f63d0 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.743+0000 7f2c27fff700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 0x7f2c18033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:09.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.743+0000 7f2c27fff700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 0x7f2c18033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c140096a0 tx=0x7f2c14006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:09.737 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.747+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2c1c00fe40 con 0x7f2c280f63d0 2024-02-20T13:42:09.942 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.952+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c28108010 con 0x7f2c280f63d0 2024-02-20T13:42:09.943 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.953+0000 7f2c257fa700 1 -- 172.21.15.192:0/3636923937 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c1c0170c0 con 0x7f2c280f63d0 2024-02-20T13:42:09.943 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:09.943 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:09.944 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 msgr2=0x7f2c18033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 0x7f2c18033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c140096a0 tx=0x7f2c14006b40).stop 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 msgr2=0x7f2c28106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2c1c0089f0 tx=0x7f2c1c00b040).stop 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 shutdown_connections 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2c18031090 0x7f2c18033550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 --2- 172.21.15.192:0/3636923937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c280f63d0 0x7f2c28106640 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 >> 172.21.15.192:0/3636923937 conn(0x7f2c280f1b50 msgr2=0x7f2c280f2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 shutdown_connections 2024-02-20T13:42:09.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:09.955+0000 7f2c2d9fc700 1 -- 172.21.15.192:0/3636923937 wait complete. 2024-02-20T13:42:09.946 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:10.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:09.954530+0000 mon.smithi158 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.192:0/3636923937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:11.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:09.332425+0000 mgr.smithi158.gmoobf (mgr.14156) 39 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:11.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:11.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 -- 172.21.15.192:0/1488389289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 msgr2=0x7f54440f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 --2- 172.21.15.192:0/1488389289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f54440f91c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5438009fb0 tx=0x7f5438009b30).stop 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 -- 172.21.15.192:0/1488389289 shutdown_connections 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 --2- 172.21.15.192:0/1488389289 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f54440f91c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 -- 172.21.15.192:0/1488389289 >> 172.21.15.192:0/1488389289 conn(0x7f54440f0be0 msgr2=0x7f54440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 -- 172.21.15.192:0/1488389289 shutdown_connections 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 -- 172.21.15.192:0/1488389289 wait complete. 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.466+0000 7f544adc0700 1 Processor -- start 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f544adc0700 1 -- start start 2024-02-20T13:42:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f544adc0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:13.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f544adc0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444106c70 con 0x7f5444060300 2024-02-20T13:42:13.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f5448b5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:13.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f5448b5c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:60126/0 (socket says 172.21.15.192:60126) 2024-02-20T13:42:13.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f5448b5c700 1 -- 172.21.15.192:0/4294248884 learned_addr learned my addr 172.21.15.192:0/4294248884 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:13.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.467+0000 7f5448b5c700 1 -- 172.21.15.192:0/4294248884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438005040 con 0x7f5444060300 2024-02-20T13:42:13.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f5448b5c700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5438009f80 tx=0x7f543800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:13.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438009320 con 0x7f5444060300 2024-02-20T13:42:13.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5444106e70 con 0x7f5444060300 2024-02-20T13:42:13.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5438013680 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f543801eda0 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5444107300 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.468+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f5438014420 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.469+0000 7f54417fa700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 0x7f5434033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.469+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5438042460 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.469+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54440591e0 con 0x7f5444060300 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.469+0000 7f5443fff700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 0x7f5434033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.469+0000 7f5443fff700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 0x7f5434033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54300096a0 tx=0x7f5430006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:13.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.474+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5438014700 con 0x7f5444060300 2024-02-20T13:42:13.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:11.332797+0000 mgr.smithi158.gmoobf (mgr.14156) 40 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:13.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.678+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5444045bb0 con 0x7f5444060300 2024-02-20T13:42:13.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.679+0000 7f54417fa700 1 -- 172.21.15.192:0/4294248884 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f543801c320 con 0x7f5444060300 2024-02-20T13:42:13.669 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:13.669 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.680+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 msgr2=0x7f5434033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.680+0000 7f544adc0700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 0x7f5434033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54300096a0 tx=0x7f5430006b40).stop 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.680+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 msgr2=0x7f5444106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.680+0000 7f544adc0700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5438009f80 tx=0x7f543800b040).stop 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 shutdown_connections 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5434031090 0x7f5434033550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 --2- 172.21.15.192:0/4294248884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5444060300 0x7f5444106780 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 >> 172.21.15.192:0/4294248884 conn(0x7f54440f0be0 msgr2=0x7f54440f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:13.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 shutdown_connections 2024-02-20T13:42:13.672 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:13.681+0000 7f544adc0700 1 -- 172.21.15.192:0/4294248884 wait complete. 2024-02-20T13:42:13.672 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:14.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:14 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:13.680447+0000 mon.smithi158 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.192:0/4294248884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:15.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:15.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:15.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:13.333175+0000 mgr.smithi158.gmoobf (mgr.14156) 41 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:16.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 -- 172.21.15.192:0/2147430481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 msgr2=0x7efde00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:16.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 --2- 172.21.15.192:0/2147430481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde00f58a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7efddc009fb0 tx=0x7efddc009b30).stop 2024-02-20T13:42:16.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 -- 172.21.15.192:0/2147430481 shutdown_connections 2024-02-20T13:42:16.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 --2- 172.21.15.192:0/2147430481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde00f58a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 -- 172.21.15.192:0/2147430481 >> 172.21.15.192:0/2147430481 conn(0x7efde00f0be0 msgr2=0x7efde00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 -- 172.21.15.192:0/2147430481 shutdown_connections 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.959+0000 7efde78ea700 1 -- 172.21.15.192:0/2147430481 wait complete. 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.960+0000 7efde78ea700 1 Processor -- start 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.960+0000 7efde78ea700 1 -- start start 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.960+0000 7efde78ea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.960+0000 7efde78ea700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde0105be0 con 0x7efde00f5480 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde5686700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde5686700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:60146/0 (socket says 172.21.15.192:60146) 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde5686700 1 -- 172.21.15.192:0/425637539 learned_addr learned my addr 172.21.15.192:0/425637539 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde5686700 1 -- 172.21.15.192:0/425637539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc005040 con 0x7efde00f5480 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde5686700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7efddc0089f0 tx=0x7efddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efddc009320 con 0x7efde00f5480 2024-02-20T13:42:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde0105de0 con 0x7efde00f5480 2024-02-20T13:42:16.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efddc013ec0 con 0x7efde00f5480 2024-02-20T13:42:16.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efddc014d30 con 0x7efde00f5480 2024-02-20T13:42:16.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.961+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde0106270 con 0x7efde00f5480 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.962+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7efddc014420 con 0x7efde00f5480 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.962+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0045bb0 con 0x7efde00f5480 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.962+0000 7efdd5ffb700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 0x7efdcc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.963+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efddc0423d0 con 0x7efde00f5480 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.963+0000 7efde4e85700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 0x7efdcc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.963+0000 7efde4e85700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 0x7efdcc033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7efdd00096a0 tx=0x7efdd0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:16.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:16.967+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efddc01d7f0 con 0x7efde00f5480 2024-02-20T13:42:17.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.174+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efde00faf70 con 0x7efde00f5480 2024-02-20T13:42:17.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.175+0000 7efdd5ffb700 1 -- 172.21.15.192:0/425637539 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efddc00fe40 con 0x7efde00f5480 2024-02-20T13:42:17.165 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:17.165 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:17.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 msgr2=0x7efdcc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:17.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 0x7efdcc033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7efdd00096a0 tx=0x7efdd0006b40).stop 2024-02-20T13:42:17.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 msgr2=0x7efde01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7efddc0089f0 tx=0x7efddc00b040).stop 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 shutdown_connections 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7efdcc031090 0x7efdcc033550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 --2- 172.21.15.192:0/425637539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efde00f5480 0x7efde01056f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 >> 172.21.15.192:0/425637539 conn(0x7efde00f0be0 msgr2=0x7efde00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 shutdown_connections 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:17.176+0000 7efde78ea700 1 -- 172.21.15.192:0/425637539 wait complete. 2024-02-20T13:42:17.167 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:17.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:15.333586+0000 mgr.smithi158.gmoobf (mgr.14156) 42 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:18.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:17.176010+0000 mon.smithi158 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.192:0/425637539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:18.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:18.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:19.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:17.333901+0000 mgr.smithi158.gmoobf (mgr.14156) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 -- 172.21.15.192:0/2565503061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 msgr2=0x7f77ec0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 --2- 172.21.15.192:0/2565503061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec0f58a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f77e0009fb0 tx=0x7f77e0009b30).stop 2024-02-20T13:42:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 -- 172.21.15.192:0/2565503061 shutdown_connections 2024-02-20T13:42:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 --2- 172.21.15.192:0/2565503061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec0f58a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 -- 172.21.15.192:0/2565503061 >> 172.21.15.192:0/2565503061 conn(0x7f77ec0f0be0 msgr2=0x7f77ec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 -- 172.21.15.192:0/2565503061 shutdown_connections 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.600+0000 7f77f136e700 1 -- 172.21.15.192:0/2565503061 wait complete. 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77f136e700 1 Processor -- start 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77f136e700 1 -- start start 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77f136e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77f136e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec105be0 con 0x7f77ec0f5480 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:60150/0 (socket says 172.21.15.192:60150) 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.601+0000 7f77eaffd700 1 -- 172.21.15.192:0/1476589235 learned_addr learned my addr 172.21.15.192:0/1476589235 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77eaffd700 1 -- 172.21.15.192:0/1476589235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e0005040 con 0x7f77ec0f5480 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77eaffd700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f77e0000c00 tx=0x7f77e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e00095f0 con 0x7f77ec0f5480 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec105de0 con 0x7f77ec0f5480 2024-02-20T13:42:20.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f77e0007e50 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e0014e70 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.602+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec106270 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.603+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f77e0025070 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.603+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec045bb0 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.603+0000 7f77d37fe700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 0x7f77d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.603+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f77e0041db0 con 0x7f77ec0f5480 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.603+0000 7f77ea7fc700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 0x7f77d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.604+0000 7f77ea7fc700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 0x7f77d4033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f77dc0096a0 tx=0x7f77dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:20.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.608+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f77e000f4e0 con 0x7f77ec0f5480 2024-02-20T13:42:20.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.814+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77ec0faf70 con 0x7f77ec0f5480 2024-02-20T13:42:20.804 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.814+0000 7f77d37fe700 1 -- 172.21.15.192:0/1476589235 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f77e000f300 con 0x7f77ec0f5480 2024-02-20T13:42:20.805 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:20.805 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 msgr2=0x7f77d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 0x7f77d4033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f77dc0096a0 tx=0x7f77dc006b40).stop 2024-02-20T13:42:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 msgr2=0x7f77ec1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f77e0000c00 tx=0x7f77e000b040).stop 2024-02-20T13:42:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 shutdown_connections 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f77d4031090 0x7f77d4033550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 --2- 172.21.15.192:0/1476589235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77ec0f5480 0x7f77ec1056f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 >> 172.21.15.192:0/1476589235 conn(0x7f77ec0f0be0 msgr2=0x7f77ec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 shutdown_connections 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:20.816+0000 7f77f136e700 1 -- 172.21.15.192:0/1476589235 wait complete. 2024-02-20T13:42:20.807 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:21.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:19.334180+0000 mgr.smithi158.gmoobf (mgr.14156) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:21.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:20.815883+0000 mon.smithi158 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.192:0/1476589235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:22.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:22.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:21.334439+0000 mgr.smithi158.gmoobf (mgr.14156) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.201+0000 7ffb4ce94700 1 -- 172.21.15.192:0/1243712466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 msgr2=0x7ffb480f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.201+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/1243712466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb480f58a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ffb30009fb0 tx=0x7ffb30009b30).stop 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 -- 172.21.15.192:0/1243712466 shutdown_connections 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/1243712466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb480f58a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 -- 172.21.15.192:0/1243712466 >> 172.21.15.192:0/1243712466 conn(0x7ffb480f0be0 msgr2=0x7ffb480f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 -- 172.21.15.192:0/1243712466 shutdown_connections 2024-02-20T13:42:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 -- 172.21.15.192:0/1243712466 wait complete. 2024-02-20T13:42:24.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 Processor -- start 2024-02-20T13:42:24.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 -- start start 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.202+0000 7ffb4ce94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4ce94700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb48105be0 con 0x7ffb480f5480 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:35344/0 (socket says 172.21.15.192:35344) 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4659c700 1 -- 172.21.15.192:0/549389624 learned_addr learned my addr 172.21.15.192:0/549389624 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4659c700 1 -- 172.21.15.192:0/549389624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30005040 con 0x7ffb480f5480 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4659c700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ffb30000c00 tx=0x7ffb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb300095f0 con 0x7ffb480f5480 2024-02-20T13:42:24.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb48105de0 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ffb30007e50 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb30014e70 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.203+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb48106270 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.204+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ffb300146b0 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.204+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb48045bb0 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.204+0000 7ffb3effd700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 0x7ffb340335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.205+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ffb30041d10 con 0x7ffb480f5480 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.205+0000 7ffb45d9b700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 0x7ffb340335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:24.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.205+0000 7ffb45d9b700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 0x7ffb340335a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffb380096a0 tx=0x7ffb38006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:24.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.209+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ffb3001b4a0 con 0x7ffb480f5480 2024-02-20T13:42:24.404 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.414+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb480faf70 con 0x7ffb480f5480 2024-02-20T13:42:24.404 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.414+0000 7ffb3effd700 1 -- 172.21.15.192:0/549389624 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb3001b2c0 con 0x7ffb480f5480 2024-02-20T13:42:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:24.405 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 msgr2=0x7ffb340335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 0x7ffb340335a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffb380096a0 tx=0x7ffb38006b40).stop 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 msgr2=0x7ffb481056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ffb30000c00 tx=0x7ffb3000b040).stop 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 shutdown_connections 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ffb340310e0 0x7ffb340335a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 --2- 172.21.15.192:0/549389624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb480f5480 0x7ffb481056f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 >> 172.21.15.192:0/549389624 conn(0x7ffb480f0be0 msgr2=0x7ffb480f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 shutdown_connections 2024-02-20T13:42:24.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:24.416+0000 7ffb4ce94700 1 -- 172.21.15.192:0/549389624 wait complete. 2024-02-20T13:42:24.407 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:25.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:23.334823+0000 mgr.smithi158.gmoobf (mgr.14156) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:25.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:24.415999+0000 mon.smithi158 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.192:0/549389624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:25.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:25.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:27.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:25.335249+0000 mgr.smithi158.gmoobf (mgr.14156) 47 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:27.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 -- 172.21.15.192:0/1701035839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 msgr2=0x7f26e80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:27.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 --2- 172.21.15.192:0/1701035839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e80f58a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f26dc009fb0 tx=0x7f26dc009b30).stop 2024-02-20T13:42:27.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 -- 172.21.15.192:0/1701035839 shutdown_connections 2024-02-20T13:42:27.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 --2- 172.21.15.192:0/1701035839 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e80f58a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:27.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 -- 172.21.15.192:0/1701035839 >> 172.21.15.192:0/1701035839 conn(0x7f26e80f0be0 msgr2=0x7f26e80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 -- 172.21.15.192:0/1701035839 shutdown_connections 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.709+0000 7f26eef87700 1 -- 172.21.15.192:0/1701035839 wait complete. 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26eef87700 1 Processor -- start 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26eef87700 1 -- start start 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26eef87700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26eef87700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8106ce0 con 0x7f26e80f5480 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26ecd23700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26ecd23700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:35362/0 (socket says 172.21.15.192:35362) 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26ecd23700 1 -- 172.21.15.192:0/3466938748 learned_addr learned my addr 172.21.15.192:0/3466938748 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.710+0000 7f26ecd23700 1 -- 172.21.15.192:0/3466938748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc005040 con 0x7f26e80f5480 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26ecd23700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f26dc0089f0 tx=0x7f26dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc009320 con 0x7f26e80f5480 2024-02-20T13:42:27.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f26dc013ec0 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e8106ee0 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc014d30 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e8107370 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.711+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f26dc0263a0 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.712+0000 7f26e57fa700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 0x7f26d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.712+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e8045bb0 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.712+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26dc042440 con 0x7f26e80f5480 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.712+0000 7f26e7fff700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 0x7f26d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.712+0000 7f26e7fff700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 0x7f26d8033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f26d40096a0 tx=0x7f26d4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:27.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.717+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f26dc012510 con 0x7f26e80f5480 2024-02-20T13:42:27.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.921+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26e80fb060 con 0x7f26e80f5480 2024-02-20T13:42:27.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.922+0000 7f26e57fa700 1 -- 172.21.15.192:0/3466938748 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26dc012330 con 0x7f26e80f5480 2024-02-20T13:42:27.911 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:27.911 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:27.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 msgr2=0x7f26d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:27.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 0x7f26d8033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f26d40096a0 tx=0x7f26d4006b40).stop 2024-02-20T13:42:27.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 msgr2=0x7f26e81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:27.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f26dc0089f0 tx=0x7f26dc00b040).stop 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 shutdown_connections 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26d8031090 0x7f26d8033550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 --2- 172.21.15.192:0/3466938748 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26e80f5480 0x7f26e81067f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 >> 172.21.15.192:0/3466938748 conn(0x7f26e80f0be0 msgr2=0x7f26e80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 shutdown_connections 2024-02-20T13:42:27.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:27.923+0000 7f26eef87700 1 -- 172.21.15.192:0/3466938748 wait complete. 2024-02-20T13:42:27.914 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:28.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:27.921925+0000 mon.smithi158 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.192:0/3466938748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:29.504 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:29.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:29.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:27.335612+0000 mgr.smithi158.gmoobf (mgr.14156) 48 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:31.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- 172.21.15.192:0/2591061924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 msgr2=0x7f68080f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 --2- 172.21.15.192:0/2591061924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f68080f69b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f67f8009fb0 tx=0x7f67f8009b30).stop 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- 172.21.15.192:0/2591061924 shutdown_connections 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 --2- 172.21.15.192:0/2591061924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f68080f69b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- 172.21.15.192:0/2591061924 >> 172.21.15.192:0/2591061924 conn(0x7f68080f1cf0 msgr2=0x7f68080f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- 172.21.15.192:0/2591061924 shutdown_connections 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- 172.21.15.192:0/2591061924 wait complete. 2024-02-20T13:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 Processor -- start 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- start start 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.274+0000 7f680de7f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6808106c70 con 0x7f68080f6590 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f68077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f68077fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:35380/0 (socket says 172.21.15.192:35380) 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f68077fe700 1 -- 172.21.15.192:0/1620488249 learned_addr learned my addr 172.21.15.192:0/1620488249 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f68077fe700 1 -- 172.21.15.192:0/1620488249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8005040 con 0x7f68080f6590 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f68077fe700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f67f80089f0 tx=0x7f67f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67f8009320 con 0x7f68080f6590 2024-02-20T13:42:31.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6808106e70 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f67f8013ec0 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67f8014d30 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.275+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6808107300 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.276+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f67f8014420 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.276+0000 7f67fffff700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 0x7f67f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.276+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f67f8042450 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.276+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6808045bb0 con 0x7f68080f6590 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.276+0000 7f6806ffd700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 0x7f67f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.277+0000 7f6806ffd700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 0x7f67f4033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f67f00096a0 tx=0x7f67f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:31.270 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.281+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f67f8014750 con 0x7f68080f6590 2024-02-20T13:42:31.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.486+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6808107fb0 con 0x7f68080f6590 2024-02-20T13:42:31.475 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.486+0000 7f67fffff700 1 -- 172.21.15.192:0/1620488249 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67f801d7f0 con 0x7f68080f6590 2024-02-20T13:42:31.475 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:31.475 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 msgr2=0x7f67f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 0x7f67f4033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f67f00096a0 tx=0x7f67f0006b40).stop 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 msgr2=0x7f6808106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f67f80089f0 tx=0x7f67f800b040).stop 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 shutdown_connections 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f67f4031090 0x7f67f4033550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 --2- 172.21.15.192:0/1620488249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f68080f6590 0x7f6808106780 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 >> 172.21.15.192:0/1620488249 conn(0x7f68080f1cf0 msgr2=0x7f68080f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 shutdown_connections 2024-02-20T13:42:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:31.488+0000 7f680de7f700 1 -- 172.21.15.192:0/1620488249 wait complete. 2024-02-20T13:42:31.478 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:31.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:29.335930+0000 mgr.smithi158.gmoobf (mgr.14156) 49 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:32.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:32 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:31.486657+0000 mon.smithi158 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.192:0/1620488249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:33.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:33.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:33.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:31.336334+0000 mgr.smithi158.gmoobf (mgr.14156) 50 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- 172.21.15.192:0/1542241020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 msgr2=0x7f23080f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 --2- 172.21.15.192:0/1542241020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f23080f69b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f22f8004760 tx=0x7f22f8009ad0).stop 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- 172.21.15.192:0/1542241020 shutdown_connections 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 --2- 172.21.15.192:0/1542241020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f23080f69b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- 172.21.15.192:0/1542241020 >> 172.21.15.192:0/1542241020 conn(0x7f23080f1cd0 msgr2=0x7f23080f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- 172.21.15.192:0/1542241020 shutdown_connections 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- 172.21.15.192:0/1542241020 wait complete. 2024-02-20T13:42:35.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 Processor -- start 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- start start 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.123+0000 7f230cc84700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308069f00 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:46492/0 (socket says 172.21.15.192:46492) 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230659c700 1 -- 172.21.15.192:0/3286207111 learned_addr learned my addr 172.21.15.192:0/3286207111 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230659c700 1 -- 172.21.15.192:0/3286207111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f8009d70 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230659c700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f22f8004170 tx=0x7f22f8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f800e040 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22f8008ef0 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230806a0a0 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f8014c90 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.124+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230806a530 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.125+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f22f8014420 con 0x7f23080f6590 2024-02-20T13:42:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.125+0000 7f22f6ffd700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 0x7f22f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:35.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.125+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f22f8042290 con 0x7f23080f6590 2024-02-20T13:42:35.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.125+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23080591e0 con 0x7f23080f6590 2024-02-20T13:42:35.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.125+0000 7f2305d9b700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 0x7f22f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:35.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.126+0000 7f2305d9b700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 0x7f22f0033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:35.118 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.130+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f22f8026300 con 0x7f23080f6590 2024-02-20T13:42:35.326 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.337+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f230806ac30 con 0x7f23080f6590 2024-02-20T13:42:35.326 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.337+0000 7f22f6ffd700 1 -- 172.21.15.192:0/3286207111 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22f8012b20 con 0x7f23080f6590 2024-02-20T13:42:35.326 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:35.326 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 msgr2=0x7f22f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 0x7f22f0033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).stop 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 msgr2=0x7f230806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f22f8004170 tx=0x7f22f8009ad0).stop 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 shutdown_connections 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22f0031090 0x7f22f0033550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 --2- 172.21.15.192:0/3286207111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23080f6590 0x7f230806ba80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 >> 172.21.15.192:0/3286207111 conn(0x7f23080f1cd0 msgr2=0x7f23080f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:35.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 shutdown_connections 2024-02-20T13:42:35.329 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:35.339+0000 7f230cc84700 1 -- 172.21.15.192:0/3286207111 wait complete. 2024-02-20T13:42:35.329 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:35.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:33.336769+0000 mgr.smithi158.gmoobf (mgr.14156) 51 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:36.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:35.337904+0000 mon.smithi158 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.192:0/3286207111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:36.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:36.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:37.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:35.337206+0000 mgr.smithi158.gmoobf (mgr.14156) 52 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 -- 172.21.15.192:0/3515043167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 msgr2=0x7f9a700f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 --2- 172.21.15.192:0/3515043167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f9200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f9a6c009fb0 tx=0x7f9a6c009b30).stop 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 -- 172.21.15.192:0/3515043167 shutdown_connections 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 --2- 172.21.15.192:0/3515043167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f9200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 -- 172.21.15.192:0/3515043167 >> 172.21.15.192:0/3515043167 conn(0x7f9a700f0be0 msgr2=0x7f9a700f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:38.947 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 -- 172.21.15.192:0/3515043167 shutdown_connections 2024-02-20T13:42:38.948 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.957+0000 7f9a78689700 1 -- 172.21.15.192:0/3515043167 wait complete. 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a78689700 1 Processor -- start 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a78689700 1 -- start start 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a78689700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a78689700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700f5100 con 0x7f9a700602c0 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a76425700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a76425700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:46520/0 (socket says 172.21.15.192:46520) 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a76425700 1 -- 172.21.15.192:0/3143205292 learned_addr learned my addr 172.21.15.192:0/3143205292 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.958+0000 7f9a76425700 1 -- 172.21.15.192:0/3143205292 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c005040 con 0x7f9a700602c0 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a76425700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9a6c009d00 tx=0x7f9a6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a6c018070 con 0x7f9a700602c0 2024-02-20T13:42:38.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a6c01d420 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a700f52a0 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a6c01c5a0 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.959+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a700f5730 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9a6c007a90 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a66ffd700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 0x7f9a5c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a700591e0 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a6c0428e0 con 0x7f9a700602c0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a75c24700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 0x7f9a5c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:38.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.960+0000 7f9a75c24700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 0x7f9a5c033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9a600096a0 tx=0x7f9a60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:38.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:38.964+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a6c027440 con 0x7f9a700602c0 2024-02-20T13:42:39.160 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.171+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a700fc070 con 0x7f9a700602c0 2024-02-20T13:42:39.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.171+0000 7f9a66ffd700 1 -- 172.21.15.192:0/3143205292 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9a6c01b370 con 0x7f9a700602c0 2024-02-20T13:42:39.161 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:39.161 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 msgr2=0x7f9a5c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 0x7f9a5c033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9a600096a0 tx=0x7f9a60006b40).stop 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 msgr2=0x7f9a700f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9a6c009d00 tx=0x7f9a6c009b30).stop 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 shutdown_connections 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a5c031090 0x7f9a5c033550 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 --2- 172.21.15.192:0/3143205292 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a700602c0 0x7f9a700f6c80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 >> 172.21.15.192:0/3143205292 conn(0x7f9a700f0be0 msgr2=0x7f9a700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 shutdown_connections 2024-02-20T13:42:39.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:39.173+0000 7f9a78689700 1 -- 172.21.15.192:0/3143205292 wait complete. 2024-02-20T13:42:39.164 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:39.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:37.337616+0000 mgr.smithi158.gmoobf (mgr.14156) 53 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:39.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:39.172756+0000 mon.smithi158 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.192:0/3143205292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:40.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:40.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:41.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:39.337889+0000 mgr.smithi158.gmoobf (mgr.14156) 54 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/1921498022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 msgr2=0x7fb6e00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/1921498022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e00f58a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb6d0009fb0 tx=0x7fb6d0009b30).stop 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/1921498022 shutdown_connections 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/1921498022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e00f58a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/1921498022 >> 172.21.15.192:0/1921498022 conn(0x7fb6e00f0be0 msgr2=0x7fb6e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:42.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/1921498022 shutdown_connections 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.660+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/1921498022 wait complete. 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e7ab3700 1 Processor -- start 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e7ab3700 1 -- start start 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e7ab3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e7ab3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0105be0 con 0x7fb6e00f5480 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e584f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e584f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54158/0 (socket says 172.21.15.192:54158) 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.661+0000 7fb6e584f700 1 -- 172.21.15.192:0/3123248303 learned_addr learned my addr 172.21.15.192:0/3123248303 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6e584f700 1 -- 172.21.15.192:0/3123248303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0005040 con 0x7fb6e00f5480 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6e584f700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb6d00089f0 tx=0x7fb6d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:42.653 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6d0009320 con 0x7fb6e00f5480 2024-02-20T13:42:42.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e0105de0 con 0x7fb6e00f5480 2024-02-20T13:42:42.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb6d0013ec0 con 0x7fb6e00f5480 2024-02-20T13:42:42.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6d0014d30 con 0x7fb6e00f5480 2024-02-20T13:42:42.654 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.662+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e0106270 con 0x7fb6e00f5480 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fb6d00095f0 con 0x7fb6e00f5480 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6d67fc700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 0x7fb6cc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e0045bb0 con 0x7fb6e00f5480 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb6d0042370 con 0x7fb6e00f5480 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6e504e700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 0x7fb6cc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:42.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.663+0000 7fb6e504e700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 0x7fb6cc0335a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb6dc0096a0 tx=0x7fb6dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:42.657 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.668+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb6d001d380 con 0x7fb6e00f5480 2024-02-20T13:42:42.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.872+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6e00fb060 con 0x7fb6e00f5480 2024-02-20T13:42:42.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.873+0000 7fb6d67fc700 1 -- 172.21.15.192:0/3123248303 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6d001d380 con 0x7fb6e00f5480 2024-02-20T13:42:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:42.863 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 msgr2=0x7fb6cc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 0x7fb6cc0335a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb6dc0096a0 tx=0x7fb6dc006b40).stop 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 msgr2=0x7fb6e01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb6d00089f0 tx=0x7fb6d000b040).stop 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 shutdown_connections 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb6cc0310e0 0x7fb6cc0335a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 --2- 172.21.15.192:0/3123248303 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6e00f5480 0x7fb6e01056f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 >> 172.21.15.192:0/3123248303 conn(0x7fb6e00f0be0 msgr2=0x7fb6e00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 shutdown_connections 2024-02-20T13:42:42.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:42.874+0000 7fb6e7ab3700 1 -- 172.21.15.192:0/3123248303 wait complete. 2024-02-20T13:42:42.865 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:43.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:41.338160+0000 mgr.smithi158.gmoobf (mgr.14156) 55 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:43.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:42.874101+0000 mon.smithi158 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.192:0/3123248303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:44.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:44.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:45.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:43.338551+0000 mgr.smithi158.gmoobf (mgr.14156) 56 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- 172.21.15.192:0/2961177842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 msgr2=0x7f85b00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/2961177842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b00f58d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f85a4009fb0 tx=0x7f85a4009b30).stop 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- 172.21.15.192:0/2961177842 shutdown_connections 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/2961177842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b00f58d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- 172.21.15.192:0/2961177842 >> 172.21.15.192:0/2961177842 conn(0x7f85b00f0c10 msgr2=0x7f85b00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- 172.21.15.192:0/2961177842 shutdown_connections 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- 172.21.15.192:0/2961177842 wait complete. 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 Processor -- start 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- start start 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.327+0000 7f85b6c0f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0060320 con 0x7f85b00f54b0 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b49ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b49ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54182/0 (socket says 172.21.15.192:54182) 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b49ab700 1 -- 172.21.15.192:0/3967629078 learned_addr learned my addr 172.21.15.192:0/3967629078 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b49ab700 1 -- 172.21.15.192:0/3967629078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a4005040 con 0x7f85b00f54b0 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b49ab700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f85a40089f0 tx=0x7f85a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:46.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4009320 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b0060520 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f85a4013ec0 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4014d30 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.328+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b0109b50 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.329+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f85a4014420 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.329+0000 7f85ad7fa700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 0x7f85a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.329+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85a4042480 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.329+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b00591e0 con 0x7f85b00f54b0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.329+0000 7f85affff700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 0x7f85a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:46.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.330+0000 7f85affff700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 0x7f85a0033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f859c0096a0 tx=0x7f859c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:46.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.334+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f85a4014750 con 0x7f85b00f54b0 2024-02-20T13:42:46.528 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.538+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85b0045bb0 con 0x7f85b00f54b0 2024-02-20T13:42:46.528 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.539+0000 7f85ad7fa700 1 -- 172.21.15.192:0/3967629078 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85a402b020 con 0x7f85b00f54b0 2024-02-20T13:42:46.528 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:46.529 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 msgr2=0x7f85a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 0x7f85a0033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f859c0096a0 tx=0x7f859c006b40).stop 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 msgr2=0x7f85b01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f85a40089f0 tx=0x7f85a400b040).stop 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 shutdown_connections 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f85a0031090 0x7f85a0033550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 --2- 172.21.15.192:0/3967629078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b00f54b0 0x7f85b01094a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.540+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 >> 172.21.15.192:0/3967629078 conn(0x7f85b00f0c10 msgr2=0x7f85b00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.541+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 shutdown_connections 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:46.541+0000 7f85b6c0f700 1 -- 172.21.15.192:0/3967629078 wait complete. 2024-02-20T13:42:46.531 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:45.339027+0000 mgr.smithi158.gmoobf (mgr.14156) 57 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:46.540298+0000 mon.smithi158 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.192:0/3967629078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:48.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:48.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:49.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:47.339340+0000 mgr.smithi158.gmoobf (mgr.14156) 58 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 -- 172.21.15.192:0/263805927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 msgr2=0x7f90ec0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 --2- 172.21.15.192:0/263805927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec0f58a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f90dc009fb0 tx=0x7f90dc009b30).stop 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 -- 172.21.15.192:0/263805927 shutdown_connections 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 --2- 172.21.15.192:0/263805927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec0f58a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 -- 172.21.15.192:0/263805927 >> 172.21.15.192:0/263805927 conn(0x7f90ec0f0be0 msgr2=0x7f90ec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.871+0000 7f90f18d3700 1 -- 172.21.15.192:0/263805927 shutdown_connections 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.872+0000 7f90f18d3700 1 -- 172.21.15.192:0/263805927 wait complete. 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.872+0000 7f90f18d3700 1 Processor -- start 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.872+0000 7f90f18d3700 1 -- start start 2024-02-20T13:42:49.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.872+0000 7f90f18d3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.872+0000 7f90f18d3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec105be0 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90eaffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54208/0 (socket says 172.21.15.192:54208) 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90eaffd700 1 -- 172.21.15.192:0/4201731659 learned_addr learned my addr 172.21.15.192:0/4201731659 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90eaffd700 1 -- 172.21.15.192:0/4201731659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc005040 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90eaffd700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f90dc000c00 tx=0x7f90dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc0095f0 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f90dc007e50 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec105de0 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc014e70 con 0x7f90ec0f5480 2024-02-20T13:42:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.873+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec106270 con 0x7f90ec0f5480 2024-02-20T13:42:49.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.874+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f90dc025070 con 0x7f90ec0f5480 2024-02-20T13:42:49.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.874+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec045bb0 con 0x7f90ec0f5480 2024-02-20T13:42:49.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.874+0000 7f90d37fe700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 0x7f90d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:49.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.874+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f90dc041cf0 con 0x7f90ec0f5480 2024-02-20T13:42:49.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.875+0000 7f90ea7fc700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 0x7f90d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:49.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.875+0000 7f90ea7fc700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 0x7f90d4033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f90e00096a0 tx=0x7f90e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:49.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:49.879+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f90dc00f4e0 con 0x7f90ec0f5480 2024-02-20T13:42:50.076 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.087+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90ec0faf70 con 0x7f90ec0f5480 2024-02-20T13:42:50.077 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.087+0000 7f90d37fe700 1 -- 172.21.15.192:0/4201731659 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f90dc00f300 con 0x7f90ec0f5480 2024-02-20T13:42:50.077 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:50.077 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:50.078 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 msgr2=0x7f90d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 0x7f90d4033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f90e00096a0 tx=0x7f90e0006b40).stop 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 msgr2=0x7f90ec1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f90dc000c00 tx=0x7f90dc00b040).stop 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 shutdown_connections 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f90d4031090 0x7f90d4033550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 --2- 172.21.15.192:0/4201731659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90ec0f5480 0x7f90ec1056f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 >> 172.21.15.192:0/4201731659 conn(0x7f90ec0f0be0 msgr2=0x7f90ec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 shutdown_connections 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:50.089+0000 7f90f18d3700 1 -- 172.21.15.192:0/4201731659 wait complete. 2024-02-20T13:42:50.079 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:50.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:50.088975+0000 mon.smithi158 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.192:0/4201731659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:51.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:49.339737+0000 mgr.smithi158.gmoobf (mgr.14156) 59 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:51.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:51.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:53.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:51.340122+0000 mgr.smithi158.gmoobf (mgr.14156) 60 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 -- 172.21.15.192:0/1402453636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 msgr2=0x7f76680f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 --2- 172.21.15.192:0/1402453636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f76680f69b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f765c009fb0 tx=0x7f765c009b30).stop 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 -- 172.21.15.192:0/1402453636 shutdown_connections 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 --2- 172.21.15.192:0/1402453636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f76680f69b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 -- 172.21.15.192:0/1402453636 >> 172.21.15.192:0/1402453636 conn(0x7f76680f1cf0 msgr2=0x7f76680f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 -- 172.21.15.192:0/1402453636 shutdown_connections 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.591+0000 7f766ea7e700 1 -- 172.21.15.192:0/1402453636 wait complete. 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766ea7e700 1 Processor -- start 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766ea7e700 1 -- start start 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766ea7e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766ea7e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668106c70 con 0x7f76680f6590 2024-02-20T13:42:53.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766c81a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:53.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766c81a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:34402/0 (socket says 172.21.15.192:34402) 2024-02-20T13:42:53.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.592+0000 7f766c81a700 1 -- 172.21.15.192:0/3266325414 learned_addr learned my addr 172.21.15.192:0/3266325414 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f766c81a700 1 -- 172.21.15.192:0/3266325414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c005040 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f766c81a700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f765c0089f0 tx=0x7f765c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c009320 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f765c013ec0 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7668106e70 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c014d30 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.593+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7668107300 con 0x7f76680f6590 2024-02-20T13:42:53.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f765c0095f0 con 0x7f76680f6590 2024-02-20T13:42:53.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f76657fa700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 0x7f7658033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:53.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7668045bb0 con 0x7f76680f6590 2024-02-20T13:42:53.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f765c0423c0 con 0x7f76680f6590 2024-02-20T13:42:53.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f7667fff700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 0x7f7658033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:53.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.594+0000 7f7667fff700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 0x7f7658033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f76540096a0 tx=0x7f7654006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:53.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.599+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f765c01d380 con 0x7f76680f6590 2024-02-20T13:42:53.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.804+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7668108180 con 0x7f76680f6590 2024-02-20T13:42:53.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.805+0000 7f76657fa700 1 -- 172.21.15.192:0/3266325414 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f765c01d380 con 0x7f76680f6590 2024-02-20T13:42:53.795 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:53.795 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:53.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 msgr2=0x7f7658033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:53.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 0x7f7658033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f76540096a0 tx=0x7f7654006b40).stop 2024-02-20T13:42:53.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 msgr2=0x7f7668106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f765c0089f0 tx=0x7f765c00b040).stop 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 shutdown_connections 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7658031090 0x7f7658033550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 --2- 172.21.15.192:0/3266325414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680f6590 0x7f7668106780 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 >> 172.21.15.192:0/3266325414 conn(0x7f76680f1cf0 msgr2=0x7f76680f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 shutdown_connections 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:53.807+0000 7f766ea7e700 1 -- 172.21.15.192:0/3266325414 wait complete. 2024-02-20T13:42:53.797 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:54.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:54 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:53.806306+0000 mon.smithi158 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.192:0/3266325414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:55.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:55.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:42:55.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:53.340463+0000 mgr.smithi158.gmoobf (mgr.14156) 61 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:56.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:55.459698+0000 mon.smithi158 (mon.0) 133 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:42:56.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:55.460478+0000 mon.smithi158 (mon.0) 134 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:42:57.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 -- 172.21.15.192:0/2000366854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 msgr2=0x7f7e3c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 --2- 172.21.15.192:0/2000366854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c0f69a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7e30009fb0 tx=0x7f7e30009b30).stop 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 -- 172.21.15.192:0/2000366854 shutdown_connections 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 --2- 172.21.15.192:0/2000366854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c0f69a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 -- 172.21.15.192:0/2000366854 >> 172.21.15.192:0/2000366854 conn(0x7f7e3c0f1ce0 msgr2=0x7f7e3c0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 -- 172.21.15.192:0/2000366854 shutdown_connections 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.187+0000 7f7e4465e700 1 -- 172.21.15.192:0/2000366854 wait complete. 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e4465e700 1 Processor -- start 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e4465e700 1 -- start start 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e4465e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e4465e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c106ca0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e423fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e423fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:34412/0 (socket says 172.21.15.192:34412) 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.188+0000 7f7e423fa700 1 -- 172.21.15.192:0/3680195013 learned_addr learned my addr 172.21.15.192:0/3680195013 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e423fa700 1 -- 172.21.15.192:0/3680195013 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30005040 con 0x7f7e3c0f6580 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e423fa700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7e300089f0 tx=0x7f7e3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e30009320 con 0x7f7e3c0f6580 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c106ea0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7e30013ec0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e30014d30 con 0x7f7e3c0f6580 2024-02-20T13:42:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.189+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c107330 con 0x7f7e3c0f6580 2024-02-20T13:42:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.190+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c045bb0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.191+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f7e300095f0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.191+0000 7f7e2effd700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 0x7f7e28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:42:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.192+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7e300425d0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.192+0000 7f7e41bf9700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 0x7f7e28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:42:57.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.192+0000 7f7e41bf9700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 0x7f7e28033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7e380096a0 tx=0x7f7e38006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:42:57.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.195+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7e3000f990 con 0x7f7e3c0f6580 2024-02-20T13:42:57.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.399+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e3c1081e0 con 0x7f7e3c0f6580 2024-02-20T13:42:57.390 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.400+0000 7f7e2effd700 1 -- 172.21.15.192:0/3680195013 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e3000f340 con 0x7f7e3c0f6580 2024-02-20T13:42:57.390 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:42:57.390 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 msgr2=0x7f7e28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 0x7f7e28033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7e380096a0 tx=0x7f7e38006b40).stop 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 msgr2=0x7f7e3c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7e300089f0 tx=0x7f7e3000b040).stop 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 shutdown_connections 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7e28031090 0x7f7e28033550 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 --2- 172.21.15.192:0/3680195013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e3c0f6580 0x7f7e3c1067b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 >> 172.21.15.192:0/3680195013 conn(0x7f7e3c0f1ce0 msgr2=0x7f7e3c0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 shutdown_connections 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:42:57.402+0000 7f7e4465e700 1 -- 172.21.15.192:0/3680195013 wait complete. 2024-02-20T13:42:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:42:57.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:55.340691+0000 mgr.smithi158.gmoobf (mgr.14156) 62 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:58.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:57.341029+0000 mgr.smithi158.gmoobf (mgr.14156) 63 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:42:58.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:42:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:42:57.401638+0000 mon.smithi158 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.192:0/3680195013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:42:58.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:42:58.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:00.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3975979824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 msgr2=0x7fb9b80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:00.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3975979824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b80f58d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb9a8009fb0 tx=0x7fb9a8009b30).stop 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3975979824 shutdown_connections 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3975979824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b80f58d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3975979824 >> 172.21.15.192:0/3975979824 conn(0x7fb9b80f0c10 msgr2=0x7fb9b80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3975979824 shutdown_connections 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.784+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3975979824 wait complete. 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9c03f3700 1 Processor -- start 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9c03f3700 1 -- start start 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9c03f3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9c03f3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8107330 con 0x7fb9b80f54b0 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9be18f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9be18f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:34436/0 (socket says 172.21.15.192:34436) 2024-02-20T13:43:00.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9be18f700 1 -- 172.21.15.192:0/3576063330 learned_addr learned my addr 172.21.15.192:0/3576063330 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.785+0000 7fb9be18f700 1 -- 172.21.15.192:0/3576063330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8005040 con 0x7fb9b80f54b0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9be18f700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb9a80089f0 tx=0x7fb9a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a8009320 con 0x7fb9b80f54b0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b8060300 con 0x7fb9b80f54b0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb9a8013ec0 con 0x7fb9b80f54b0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a8014d30 con 0x7fb9b80f54b0 2024-02-20T13:43:00.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.786+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b810be30 con 0x7fb9b80f54b0 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fb9a80263a0 con 0x7fb9b80f54b0 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9aeffd700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 0x7fb9a4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb9a8042470 con 0x7fb9b80f54b0 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b8045bb0 con 0x7fb9b80f54b0 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9bd98e700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 0x7fb9a4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:00.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.787+0000 7fb9bd98e700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 0x7fb9a4033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb9b40096a0 tx=0x7fb9b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:00.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.792+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb9a80266d0 con 0x7fb9b80f54b0 2024-02-20T13:43:00.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.997+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9b80fb090 con 0x7fb9b80f54b0 2024-02-20T13:43:00.987 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.997+0000 7fb9aeffd700 1 -- 172.21.15.192:0/3576063330 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb9a802b020 con 0x7fb9b80f54b0 2024-02-20T13:43:00.987 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:00.987 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 msgr2=0x7fb9a4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 0x7fb9a4033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb9b40096a0 tx=0x7fb9b4006b40).stop 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 msgr2=0x7fb9b810b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb9a80089f0 tx=0x7fb9a800b040).stop 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 shutdown_connections 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb9a4031090 0x7fb9a4033550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 --2- 172.21.15.192:0/3576063330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9b80f54b0 0x7fb9b810b940 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 >> 172.21.15.192:0/3576063330 conn(0x7fb9b80f0c10 msgr2=0x7fb9b80f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:00.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 shutdown_connections 2024-02-20T13:43:00.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:00.999+0000 7fb9c03f3700 1 -- 172.21.15.192:0/3576063330 wait complete. 2024-02-20T13:43:00.990 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:01.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:42:59.341321+0000 mgr.smithi158.gmoobf (mgr.14156) 64 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:01.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:00.999001+0000 mon.smithi158 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.192:0/3576063330' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:02.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:02.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:03.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:01.341632+0000 mgr.smithi158.gmoobf (mgr.14156) 65 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:04.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 -- 172.21.15.192:0/1931134301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 msgr2=0x7f21680f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 --2- 172.21.15.192:0/1931134301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21680f4c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2158009fb0 tx=0x7f2158009b30).stop 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 -- 172.21.15.192:0/1931134301 shutdown_connections 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 --2- 172.21.15.192:0/1931134301 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21680f4c60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 -- 172.21.15.192:0/1931134301 >> 172.21.15.192:0/1931134301 conn(0x7f21680f0360 msgr2=0x7f21680f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 -- 172.21.15.192:0/1931134301 shutdown_connections 2024-02-20T13:43:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.318+0000 7f216ff97700 1 -- 172.21.15.192:0/1931134301 wait complete. 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216ff97700 1 Processor -- start 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216ff97700 1 -- start start 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216ff97700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216ff97700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168106cb0 con 0x7f21680f4840 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216dd33700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216dd33700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43338/0 (socket says 172.21.15.192:43338) 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216dd33700 1 -- 172.21.15.192:0/901402025 learned_addr learned my addr 172.21.15.192:0/901402025 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.319+0000 7f216dd33700 1 -- 172.21.15.192:0/901402025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158005040 con 0x7f21680f4840 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f216dd33700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f21580089f0 tx=0x7f215800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2158009320 con 0x7f21680f4840 2024-02-20T13:43:04.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2168106eb0 con 0x7f21680f4840 2024-02-20T13:43:04.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2158013ec0 con 0x7f21680f4840 2024-02-20T13:43:04.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2158014d30 con 0x7f21680f4840 2024-02-20T13:43:04.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168107340 con 0x7f21680f4840 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.320+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f21580095f0 con 0x7f21680f4840 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.321+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168045bb0 con 0x7f21680f4840 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.321+0000 7f215e7fc700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 0x7f2154033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.321+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f21580423c0 con 0x7f21680f4840 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.321+0000 7f216d532700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 0x7f2154033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:04.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.321+0000 7f216d532700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 0x7f2154033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:04.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.326+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f215801db60 con 0x7f21680f4840 2024-02-20T13:43:04.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.532+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21681081f0 con 0x7f21680f4840 2024-02-20T13:43:04.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.533+0000 7f215e7fc700 1 -- 172.21.15.192:0/901402025 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f215801dd70 con 0x7f21680f4840 2024-02-20T13:43:04.523 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:04.523 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 msgr2=0x7f2154033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 0x7f2154033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).stop 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 msgr2=0x7f21681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f21580089f0 tx=0x7f215800b040).stop 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 shutdown_connections 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2154031090 0x7f2154033550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 --2- 172.21.15.192:0/901402025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21680f4840 0x7f21681067c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 >> 172.21.15.192:0/901402025 conn(0x7f21680f0360 msgr2=0x7f21680f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 shutdown_connections 2024-02-20T13:43:04.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:04.535+0000 7f216ff97700 1 -- 172.21.15.192:0/901402025 wait complete. 2024-02-20T13:43:04.526 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:05.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:03.341918+0000 mgr.smithi158.gmoobf (mgr.14156) 66 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:05.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:04.534560+0000 mon.smithi158 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.192:0/901402025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:06.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:06.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:07.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:05.342231+0000 mgr.smithi158.gmoobf (mgr.14156) 67 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:07.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.950+0000 7f887cf56700 1 -- 172.21.15.192:0/328199572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 msgr2=0x7f88780f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:07.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.950+0000 7f887cf56700 1 --2- 172.21.15.192:0/328199572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f88780f5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8868009fb0 tx=0x7f8868009b30).stop 2024-02-20T13:43:07.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.950+0000 7f887cf56700 1 -- 172.21.15.192:0/328199572 shutdown_connections 2024-02-20T13:43:07.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.950+0000 7f887cf56700 1 --2- 172.21.15.192:0/328199572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f88780f5700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.950+0000 7f887cf56700 1 -- 172.21.15.192:0/328199572 >> 172.21.15.192:0/328199572 conn(0x7f88780f0a40 msgr2=0x7f88780f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 -- 172.21.15.192:0/328199572 shutdown_connections 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 -- 172.21.15.192:0/328199572 wait complete. 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 Processor -- start 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 -- start start 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.951+0000 7f887cf56700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8878106b10 con 0x7f88780f52e0 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f88777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f88777fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43352/0 (socket says 172.21.15.192:43352) 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f88777fe700 1 -- 172.21.15.192:0/1121980357 learned_addr learned my addr 172.21.15.192:0/1121980357 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f88777fe700 1 -- 172.21.15.192:0/1121980357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868005040 con 0x7f88780f52e0 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f88777fe700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f88680089f0 tx=0x7f886800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8868009320 con 0x7f88780f52e0 2024-02-20T13:43:07.971 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8878106d10 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8868013ec0 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8868014d30 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.952+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88781071a0 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.953+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f88680095f0 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.953+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8878045bb0 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.953+0000 7f885ffff700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 0x7f8860033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.953+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f88680424e0 con 0x7f88780f52e0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.953+0000 7f8876ffd700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 0x7f8860033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.954+0000 7f8876ffd700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 0x7f8860033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f886c0096a0 tx=0x7f886c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:07.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:07.958+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f886801d7f0 con 0x7f88780f52e0 2024-02-20T13:43:08.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.164+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8878107ff0 con 0x7f88780f52e0 2024-02-20T13:43:08.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.166+0000 7f885ffff700 1 -- 172.21.15.192:0/1121980357 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f886801d7f0 con 0x7f88780f52e0 2024-02-20T13:43:08.154 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:08.155 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 msgr2=0x7f8860033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 0x7f8860033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f886c0096a0 tx=0x7f886c006b40).stop 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 msgr2=0x7f8878106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f88680089f0 tx=0x7f886800b040).stop 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 shutdown_connections 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8860031090 0x7f8860033550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:08.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 --2- 172.21.15.192:0/1121980357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88780f52e0 0x7f8878106620 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:08.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 >> 172.21.15.192:0/1121980357 conn(0x7f88780f0a40 msgr2=0x7f88780f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:08.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 shutdown_connections 2024-02-20T13:43:08.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:08.167+0000 7f887cf56700 1 -- 172.21.15.192:0/1121980357 wait complete. 2024-02-20T13:43:08.157 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:09.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:07.342559+0000 mgr.smithi158.gmoobf (mgr.14156) 68 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:09.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:08.166243+0000 mon.smithi158 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.192:0/1121980357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:09.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:09.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:11.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 -- 172.21.15.192:0/1049901500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 msgr2=0x7f6e800f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 --2- 172.21.15.192:0/1049901500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e800f58a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6e74009fb0 tx=0x7f6e74009b30).stop 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 -- 172.21.15.192:0/1049901500 shutdown_connections 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 --2- 172.21.15.192:0/1049901500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e800f58a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 -- 172.21.15.192:0/1049901500 >> 172.21.15.192:0/1049901500 conn(0x7f6e800f0be0 msgr2=0x7f6e800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.532+0000 7f6e876ed700 1 -- 172.21.15.192:0/1049901500 shutdown_connections 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e876ed700 1 -- 172.21.15.192:0/1049901500 wait complete. 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e876ed700 1 Processor -- start 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e876ed700 1 -- start start 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e876ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e876ed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80106cb0 con 0x7f6e800f5480 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e85489700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:11.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e85489700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43368/0 (socket says 172.21.15.192:43368) 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.533+0000 7f6e85489700 1 -- 172.21.15.192:0/2527857444 learned_addr learned my addr 172.21.15.192:0/2527857444 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e85489700 1 -- 172.21.15.192:0/2527857444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74005040 con 0x7f6e800f5480 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e85489700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6e740089f0 tx=0x7f6e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74009320 con 0x7f6e800f5480 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e80106eb0 con 0x7f6e800f5480 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6e74013ec0 con 0x7f6e800f5480 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74014d30 con 0x7f6e800f5480 2024-02-20T13:43:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.534+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e80107340 con 0x7f6e800f5480 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.535+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6e740095f0 con 0x7f6e800f5480 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.535+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e80045bb0 con 0x7f6e800f5480 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.535+0000 7f6e71ffb700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 0x7f6e6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.535+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6e74042390 con 0x7f6e800f5480 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.536+0000 7f6e84c88700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 0x7f6e6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.536+0000 7f6e84c88700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 0x7f6e6c033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:11.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.540+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6e7401d7f0 con 0x7f6e800f5480 2024-02-20T13:43:11.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:09.342871+0000 mgr.smithi158.gmoobf (mgr.14156) 69 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:11.737 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.748+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e801081f0 con 0x7f6e800f5480 2024-02-20T13:43:11.737 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.748+0000 7f6e71ffb700 1 -- 172.21.15.192:0/2527857444 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6e7401d7f0 con 0x7f6e800f5480 2024-02-20T13:43:11.738 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:11.738 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:11.739 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 msgr2=0x7f6e6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:11.739 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 0x7f6e6c033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).stop 2024-02-20T13:43:11.739 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 msgr2=0x7f6e801067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:11.739 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6e740089f0 tx=0x7f6e7400b040).stop 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 shutdown_connections 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6e6c031090 0x7f6e6c033550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 --2- 172.21.15.192:0/2527857444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e800f5480 0x7f6e801067c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 >> 172.21.15.192:0/2527857444 conn(0x7f6e800f0be0 msgr2=0x7f6e8005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 shutdown_connections 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:11.750+0000 7f6e876ed700 1 -- 172.21.15.192:0/2527857444 wait complete. 2024-02-20T13:43:11.740 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:12.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:11.749672+0000 mon.smithi158 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.192:0/2527857444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:13.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:13.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:13.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:11.343221+0000 mgr.smithi158.gmoobf (mgr.14156) 70 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:15.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 -- 172.21.15.192:0/814598308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 msgr2=0x7fc8ac0fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/814598308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac0fb5e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc8a0009fb0 tx=0x7fc8a0009b30).stop 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 -- 172.21.15.192:0/814598308 shutdown_connections 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/814598308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac0fb5e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 -- 172.21.15.192:0/814598308 >> 172.21.15.192:0/814598308 conn(0x7fc8ac0f0be0 msgr2=0x7fc8ac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 -- 172.21.15.192:0/814598308 shutdown_connections 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 -- 172.21.15.192:0/814598308 wait complete. 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.197+0000 7fc8b0c24700 1 Processor -- start 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8b0c24700 1 -- start start 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8b0c24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8b0c24700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac108e60 con 0x7fc8ac0602e0 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8aa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:15.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8aa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55884/0 (socket says 172.21.15.192:55884) 2024-02-20T13:43:15.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8aa59c700 1 -- 172.21.15.192:0/2524197407 learned_addr learned my addr 172.21.15.192:0/2524197407 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:15.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8aa59c700 1 -- 172.21.15.192:0/2524197407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a0005040 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8aa59c700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc8a0000c00 tx=0x7fc8a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a00095f0 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac109060 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc8a0007e50 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.198+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a0014e70 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.199+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac1094f0 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.199+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc8a0025070 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.199+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac045bb0 con 0x7fc8ac0602e0 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.199+0000 7fc89affd700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 0x7fc894033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:15.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.199+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc8a0041e10 con 0x7fc8ac0602e0 2024-02-20T13:43:15.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.200+0000 7fc8a9d9b700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 0x7fc894033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:15.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.200+0000 7fc8a9d9b700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 0x7fc894033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc89c0096a0 tx=0x7fc89c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:15.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.205+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc8a000f4e0 con 0x7fc8ac0602e0 2024-02-20T13:43:15.400 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.410+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8ac10a3a0 con 0x7fc8ac0602e0 2024-02-20T13:43:15.401 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.411+0000 7fc89affd700 1 -- 172.21.15.192:0/2524197407 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc8a0043070 con 0x7fc8ac0602e0 2024-02-20T13:43:15.401 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:15.402 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:15.403 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 msgr2=0x7fc894033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:15.403 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 0x7fc894033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc89c0096a0 tx=0x7fc89c006b40).stop 2024-02-20T13:43:15.404 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 msgr2=0x7fc8ac108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:15.404 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc8a0000c00 tx=0x7fc8a000b040).stop 2024-02-20T13:43:15.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 shutdown_connections 2024-02-20T13:43:15.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc894031090 0x7fc894033550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:15.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 --2- 172.21.15.192:0/2524197407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8ac0602e0 0x7fc8ac108970 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:15.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 >> 172.21.15.192:0/2524197407 conn(0x7fc8ac0f0be0 msgr2=0x7fc8ac0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:15.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 shutdown_connections 2024-02-20T13:43:15.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:15.413+0000 7fc8b0c24700 1 -- 172.21.15.192:0/2524197407 wait complete. 2024-02-20T13:43:15.406 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:15.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:13.343513+0000 mgr.smithi158.gmoobf (mgr.14156) 71 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:16.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:15.412422+0000 mon.smithi158 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.192:0/2524197407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:17.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:17.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:17.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:15.343818+0000 mgr.smithi158.gmoobf (mgr.14156) 72 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/583351653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 msgr2=0x7f2c080f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/583351653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c080f9160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f2c04009fb0 tx=0x7f2c04009b30).stop 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/583351653 shutdown_connections 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/583351653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c080f9160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/583351653 >> 172.21.15.192:0/583351653 conn(0x7f2c080f0be0 msgr2=0x7f2c080f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/583351653 shutdown_connections 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/583351653 wait complete. 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.862+0000 7f2c0f3f8700 1 Processor -- start 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0f3f8700 1 -- start start 2024-02-20T13:43:18.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0f3f8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0f3f8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08106c40 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0d194700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0d194700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55892/0 (socket says 172.21.15.192:55892) 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0d194700 1 -- 172.21.15.192:0/2018728294 learned_addr learned my addr 172.21.15.192:0/2018728294 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0d194700 1 -- 172.21.15.192:0/2018728294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c04005040 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0d194700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c04009f80 tx=0x7f2c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c04009320 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c08106e40 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2c04013680 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.863+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0401eda0 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.864+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c081072d0 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.864+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2c0401e420 con 0x7f2c08060360 2024-02-20T13:43:18.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.865+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c08045bb0 con 0x7f2c08060360 2024-02-20T13:43:18.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.865+0000 7f2bfdffb700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 0x7f2bf4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:18.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.865+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2c04042420 con 0x7f2c08060360 2024-02-20T13:43:18.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.865+0000 7f2c0c993700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 0x7f2bf4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:18.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.865+0000 7f2c0c993700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 0x7f2bf4033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2bf80096a0 tx=0x7f2bf8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:18.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:18.869+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2c0401d380 con 0x7f2c08060360 2024-02-20T13:43:19.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.076+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c08108180 con 0x7f2c08060360 2024-02-20T13:43:19.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.077+0000 7f2bfdffb700 1 -- 172.21.15.192:0/2018728294 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c0401cb10 con 0x7f2c08060360 2024-02-20T13:43:19.066 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:19.066 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 msgr2=0x7f2bf4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 0x7f2bf4033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2bf80096a0 tx=0x7f2bf8006b40).stop 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 msgr2=0x7f2c08106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c04009f80 tx=0x7f2c0400b040).stop 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 shutdown_connections 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2bf4031090 0x7f2bf4033550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 --2- 172.21.15.192:0/2018728294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c08060360 0x7f2c08106750 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 >> 172.21.15.192:0/2018728294 conn(0x7f2c080f0be0 msgr2=0x7f2c080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 shutdown_connections 2024-02-20T13:43:19.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:19.079+0000 7f2c0f3f8700 1 -- 172.21.15.192:0/2018728294 wait complete. 2024-02-20T13:43:19.069 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:19.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:17.344082+0000 mgr.smithi158.gmoobf (mgr.14156) 73 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:19.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:19.077692+0000 mon.smithi158 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.192:0/2018728294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:20.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:20.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:21.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:19.344488+0000 mgr.smithi158.gmoobf (mgr.14156) 74 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:22.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/1210964923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 msgr2=0x7fdf180f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:22.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/1210964923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf180f58a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fdf0c009fb0 tx=0x7fdf0c009b30).stop 2024-02-20T13:43:22.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/1210964923 shutdown_connections 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/1210964923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf180f58a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/1210964923 >> 172.21.15.192:0/1210964923 conn(0x7fdf180f0be0 msgr2=0x7fdf180f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/1210964923 shutdown_connections 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.422+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/1210964923 wait complete. 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.423+0000 7fdf1f0a0700 1 Processor -- start 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.423+0000 7fdf1f0a0700 1 -- start start 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.423+0000 7fdf1f0a0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:22.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.423+0000 7fdf1f0a0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18105be0 con 0x7fdf180f5480 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1ce3c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1ce3c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36678/0 (socket says 172.21.15.192:36678) 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1ce3c700 1 -- 172.21.15.192:0/2741620812 learned_addr learned my addr 172.21.15.192:0/2741620812 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1ce3c700 1 -- 172.21.15.192:0/2741620812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c005040 con 0x7fdf180f5480 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1ce3c700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fdf0c0089f0 tx=0x7fdf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0c009320 con 0x7fdf180f5480 2024-02-20T13:43:22.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf0c013ec0 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18105de0 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0c014d30 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.424+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf18106270 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.425+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdf0c014420 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.425+0000 7fdf157fa700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 0x7fdf080335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.425+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdf0c0423a0 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.425+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf18045bb0 con 0x7fdf180f5480 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.426+0000 7fdf17fff700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 0x7fdf080335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:22.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.426+0000 7fdf17fff700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 0x7fdf080335a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fdf040096a0 tx=0x7fdf04006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:22.418 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.430+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf0c014750 con 0x7fdf180f5480 2024-02-20T13:43:22.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.638+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf180faf70 con 0x7fdf180f5480 2024-02-20T13:43:22.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.639+0000 7fdf157fa700 1 -- 172.21.15.192:0/2741620812 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf180faf70 con 0x7fdf180f5480 2024-02-20T13:43:22.627 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:22.627 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:22.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.640+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 msgr2=0x7fdf080335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:22.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.640+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 0x7fdf080335a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fdf040096a0 tx=0x7fdf04006b40).stop 2024-02-20T13:43:22.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 msgr2=0x7fdf181056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:22.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fdf0c0089f0 tx=0x7fdf0c00b040).stop 2024-02-20T13:43:22.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 shutdown_connections 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdf080310e0 0x7fdf080335a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 --2- 172.21.15.192:0/2741620812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf180f5480 0x7fdf181056f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 >> 172.21.15.192:0/2741620812 conn(0x7fdf180f0be0 msgr2=0x7fdf180f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 shutdown_connections 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:22.641+0000 7fdf1f0a0700 1 -- 172.21.15.192:0/2741620812 wait complete. 2024-02-20T13:43:22.630 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:21.344851+0000 mgr.smithi158.gmoobf (mgr.14156) 75 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:22.639191+0000 mon.smithi158 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.192:0/2741620812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:24.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:24.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:25.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:23.345146+0000 mgr.smithi158.gmoobf (mgr.14156) 76 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:26.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- 172.21.15.192:0/3255077377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 msgr2=0x7fd35c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3255077377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c0f56f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd350009fb0 tx=0x7fd350009b30).stop 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- 172.21.15.192:0/3255077377 shutdown_connections 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3255077377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c0f56f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- 172.21.15.192:0/3255077377 >> 172.21.15.192:0/3255077377 conn(0x7fd35c0f0a30 msgr2=0x7fd35c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- 172.21.15.192:0/3255077377 shutdown_connections 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- 172.21.15.192:0/3255077377 wait complete. 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 Processor -- start 2024-02-20T13:43:26.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- start start 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.077+0000 7fd361ed0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c105a00 con 0x7fd35c0f52d0 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd360ece700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd360ece700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36694/0 (socket says 172.21.15.192:36694) 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd360ece700 1 -- 172.21.15.192:0/3568678492 learned_addr learned my addr 172.21.15.192:0/3568678492 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd360ece700 1 -- 172.21.15.192:0/3568678492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd350005040 con 0x7fd35c0f52d0 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd360ece700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd350000c00 tx=0x7fd35000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3500095f0 con 0x7fd35c0f52d0 2024-02-20T13:43:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c105c00 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd350007e50 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd350014e70 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.078+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c106090 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.079+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fd350025070 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.079+0000 7fd3597fa700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 0x7fd34c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.079+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd350041e40 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.079+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c0591e0 con 0x7fd35c0f52d0 2024-02-20T13:43:26.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.079+0000 7fd35bfff700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 0x7fd34c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:26.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.080+0000 7fd35bfff700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 0x7fd34c033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd3480096a0 tx=0x7fd348006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:26.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.084+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd350026350 con 0x7fd35c0f52d0 2024-02-20T13:43:26.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.290+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd35c045bb0 con 0x7fd35c0f52d0 2024-02-20T13:43:26.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.291+0000 7fd3597fa700 1 -- 172.21.15.192:0/3568678492 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd35002c020 con 0x7fd35c0f52d0 2024-02-20T13:43:26.280 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:26.280 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:26.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 msgr2=0x7fd34c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:26.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 0x7fd34c033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd3480096a0 tx=0x7fd348006b40).stop 2024-02-20T13:43:26.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 msgr2=0x7fd35c105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:26.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd350000c00 tx=0x7fd35000b040).stop 2024-02-20T13:43:26.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 shutdown_connections 2024-02-20T13:43:26.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd34c031090 0x7fd34c033550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:26.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 --2- 172.21.15.192:0/3568678492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd35c0f52d0 0x7fd35c105510 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:26.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 >> 172.21.15.192:0/3568678492 conn(0x7fd35c0f0a30 msgr2=0x7fd35c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:26.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 shutdown_connections 2024-02-20T13:43:26.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:26.293+0000 7fd361ed0700 1 -- 172.21.15.192:0/3568678492 wait complete. 2024-02-20T13:43:26.285 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:27.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:25.345526+0000 mgr.smithi158.gmoobf (mgr.14156) 77 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:27.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:26.291573+0000 mon.smithi158 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.192:0/3568678492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:27.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:27.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:29.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:27.345916+0000 mgr.smithi158.gmoobf (mgr.14156) 78 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 -- 172.21.15.192:0/445660834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 msgr2=0x7f44940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 --2- 172.21.15.192:0/445660834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940f58a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f447c009fb0 tx=0x7f447c009b30).stop 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 -- 172.21.15.192:0/445660834 shutdown_connections 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 --2- 172.21.15.192:0/445660834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940f58a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 -- 172.21.15.192:0/445660834 >> 172.21.15.192:0/445660834 conn(0x7f44940f0be0 msgr2=0x7f44940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 -- 172.21.15.192:0/445660834 shutdown_connections 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 -- 172.21.15.192:0/445660834 wait complete. 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.864+0000 7f4499f56700 1 Processor -- start 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f4499f56700 1 -- start start 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f4499f56700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f4499f56700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44940fb8e0 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f44937fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f44937fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36716/0 (socket says 172.21.15.192:36716) 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f44937fe700 1 -- 172.21.15.192:0/3335185073 learned_addr learned my addr 172.21.15.192:0/3335185073 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f44937fe700 1 -- 172.21.15.192:0/3335185073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c005040 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f44937fe700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f447c0089f0 tx=0x7f447c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f447c009320 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f447c013ec0 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44940fba80 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.865+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f447c014d30 con 0x7f44940f5480 2024-02-20T13:43:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.866+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44940fbf10 con 0x7f44940f5480 2024-02-20T13:43:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.866+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f447c014420 con 0x7f44940f5480 2024-02-20T13:43:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.866+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44940591e0 con 0x7f44940f5480 2024-02-20T13:43:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.866+0000 7f448bfff700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 0x7f44800335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.867+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f447c042450 con 0x7f44940f5480 2024-02-20T13:43:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.867+0000 7f4492ffd700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 0x7f44800335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:29.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.867+0000 7f4492ffd700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 0x7f44800335a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f44840096a0 tx=0x7f4484006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:29.871+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f447c0268c0 con 0x7f44940f5480 2024-02-20T13:43:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.077+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44940faf70 con 0x7f44940f5480 2024-02-20T13:43:30.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.078+0000 7f448bfff700 1 -- 172.21.15.192:0/3335185073 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f447c01d7f0 con 0x7f44940f5480 2024-02-20T13:43:30.067 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:30.067 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:30.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 msgr2=0x7f44800335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 0x7f44800335a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f44840096a0 tx=0x7f4484006b40).stop 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 msgr2=0x7f44940fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f447c0089f0 tx=0x7f447c00b040).stop 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 shutdown_connections 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f44800310e0 0x7f44800335a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 --2- 172.21.15.192:0/3335185073 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44940f5480 0x7f44940fd460 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 >> 172.21.15.192:0/3335185073 conn(0x7f44940f0be0 msgr2=0x7f449405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 shutdown_connections 2024-02-20T13:43:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:30.079+0000 7f4499f56700 1 -- 172.21.15.192:0/3335185073 wait complete. 2024-02-20T13:43:30.070 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:30.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:30.078961+0000 mon.smithi158 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.192:0/3335185073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:31.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:29.346331+0000 mgr.smithi158.gmoobf (mgr.14156) 79 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:31.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:31.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- 172.21.15.192:0/3742209371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 msgr2=0x7f28a80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 --2- 172.21.15.192:0/3742209371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a80f56f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f289c009fb0 tx=0x7f289c009b30).stop 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- 172.21.15.192:0/3742209371 shutdown_connections 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 --2- 172.21.15.192:0/3742209371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a80f56f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- 172.21.15.192:0/3742209371 >> 172.21.15.192:0/3742209371 conn(0x7f28a80f0a30 msgr2=0x7f28a80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:33.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- 172.21.15.192:0/3742209371 shutdown_connections 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- 172.21.15.192:0/3742209371 wait complete. 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 Processor -- start 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- start start 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.537+0000 7f28adb31700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8106af0 con 0x7f28a80f52d0 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28acb2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28acb2f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41148/0 (socket says 172.21.15.192:41148) 2024-02-20T13:43:33.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28acb2f700 1 -- 172.21.15.192:0/2639067713 learned_addr learned my addr 172.21.15.192:0/2639067713 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28acb2f700 1 -- 172.21.15.192:0/2639067713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c005040 con 0x7f28a80f52d0 2024-02-20T13:43:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28acb2f700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f289c000c00 tx=0x7f289c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f289c0095f0 con 0x7f28a80f52d0 2024-02-20T13:43:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a8106cf0 con 0x7f28a80f52d0 2024-02-20T13:43:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f289c007e50 con 0x7f28a80f52d0 2024-02-20T13:43:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f289c014e70 con 0x7f28a80f52d0 2024-02-20T13:43:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.538+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a8107180 con 0x7f28a80f52d0 2024-02-20T13:43:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.539+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f289c025070 con 0x7f28a80f52d0 2024-02-20T13:43:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.539+0000 7f28a57fa700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 0x7f2898033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.539+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f289c041d10 con 0x7f28a80f52d0 2024-02-20T13:43:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.539+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a80591e0 con 0x7f28a80f52d0 2024-02-20T13:43:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.539+0000 7f28a7fff700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 0x7f2898033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.540+0000 7f28a7fff700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 0x7f2898033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28940096a0 tx=0x7f2894006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:33.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.544+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f289c0079c0 con 0x7f28a80f52d0 2024-02-20T13:43:33.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:31.346607+0000 mgr.smithi158.gmoobf (mgr.14156) 80 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:33.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.751+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28a8045bb0 con 0x7f28a80f52d0 2024-02-20T13:43:33.742 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.752+0000 7f28a57fa700 1 -- 172.21.15.192:0/2639067713 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f289c02d090 con 0x7f28a80f52d0 2024-02-20T13:43:33.742 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:33.743 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 msgr2=0x7f2898033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 0x7f2898033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28940096a0 tx=0x7f2894006b40).stop 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 msgr2=0x7f28a8106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f289c000c00 tx=0x7f289c00b040).stop 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 shutdown_connections 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2898031090 0x7f2898033550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 --2- 172.21.15.192:0/2639067713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28a80f52d0 0x7f28a8106600 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 >> 172.21.15.192:0/2639067713 conn(0x7f28a80f0a30 msgr2=0x7f28a805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 shutdown_connections 2024-02-20T13:43:33.745 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:33.754+0000 7f28adb31700 1 -- 172.21.15.192:0/2639067713 wait complete. 2024-02-20T13:43:33.745 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:34.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:33.753233+0000 mon.smithi158 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.192:0/2639067713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:35.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:35.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:35.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:33.346933+0000 mgr.smithi158.gmoobf (mgr.14156) 81 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:37.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.192+0000 7f041197c700 1 -- 172.21.15.192:0/4188785106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 msgr2=0x7f040c0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.192+0000 7f041197c700 1 --2- 172.21.15.192:0/4188785106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c0f6800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f0400009fb0 tx=0x7f0400009b30).stop 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.192+0000 7f041197c700 1 -- 172.21.15.192:0/4188785106 shutdown_connections 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.192+0000 7f041197c700 1 --2- 172.21.15.192:0/4188785106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c0f6800 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.192+0000 7f041197c700 1 -- 172.21.15.192:0/4188785106 >> 172.21.15.192:0/4188785106 conn(0x7f040c0f1b40 msgr2=0x7f040c0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 -- 172.21.15.192:0/4188785106 shutdown_connections 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 -- 172.21.15.192:0/4188785106 wait complete. 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 Processor -- start 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 -- start start 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.193+0000 7f041197c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c106b10 con 0x7f040c0f63e0 2024-02-20T13:43:37.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041097a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041097a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41172/0 (socket says 172.21.15.192:41172) 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041097a700 1 -- 172.21.15.192:0/2820135368 learned_addr learned my addr 172.21.15.192:0/2820135368 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041097a700 1 -- 172.21.15.192:0/2820135368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400005040 con 0x7f040c0f63e0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041097a700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f04000089f0 tx=0x7f040000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0400009320 con 0x7f040c0f63e0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c106d10 con 0x7f040c0f63e0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0400013ec0 con 0x7f040c0f63e0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0400014d30 con 0x7f040c0f63e0 2024-02-20T13:43:37.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.194+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c1071a0 con 0x7f040c0f63e0 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.195+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f04000263a0 con 0x7f040c0f63e0 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.195+0000 7f04097fa700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 0x7f03fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.195+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0400042380 con 0x7f040c0f63e0 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.195+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c045bb0 con 0x7f040c0f63e0 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.196+0000 7f040bfff700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 0x7f03fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.196+0000 7f040bfff700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 0x7f03fc033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f03f80096a0 tx=0x7f03f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:37.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.200+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f04000266d0 con 0x7f040c0f63e0 2024-02-20T13:43:37.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.406+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f040c107f20 con 0x7f040c0f63e0 2024-02-20T13:43:37.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.407+0000 7f04097fa700 1 -- 172.21.15.192:0/2820135368 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f040002b020 con 0x7f040c0f63e0 2024-02-20T13:43:37.396 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:37.396 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.408+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 msgr2=0x7f03fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.408+0000 7f041197c700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 0x7f03fc033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f03f80096a0 tx=0x7f03f8006b40).stop 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.408+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 msgr2=0x7f040c106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.408+0000 7f041197c700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f04000089f0 tx=0x7f040000b040).stop 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 shutdown_connections 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f03fc031090 0x7f03fc033550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 --2- 172.21.15.192:0/2820135368 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040c0f63e0 0x7f040c106620 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 >> 172.21.15.192:0/2820135368 conn(0x7f040c0f1b40 msgr2=0x7f040c0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 shutdown_connections 2024-02-20T13:43:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:37.409+0000 7f041197c700 1 -- 172.21.15.192:0/2820135368 wait complete. 2024-02-20T13:43:37.399 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:37.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:35.347283+0000 mgr.smithi158.gmoobf (mgr.14156) 82 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:38.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:37.408576+0000 mon.smithi158 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.192:0/2820135368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:39.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:39.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:39.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:37.347707+0000 mgr.smithi158.gmoobf (mgr.14156) 83 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:40.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/3173300214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 msgr2=0x7f5fdc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:40.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/3173300214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc0f58a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5fd4009fb0 tx=0x7f5fd4009b30).stop 2024-02-20T13:43:40.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/3173300214 shutdown_connections 2024-02-20T13:43:40.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/3173300214 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc0f58a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:40.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/3173300214 >> 172.21.15.192:0/3173300214 conn(0x7f5fdc0f0be0 msgr2=0x7f5fdc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:40.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/3173300214 shutdown_connections 2024-02-20T13:43:40.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.789+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/3173300214 wait complete. 2024-02-20T13:43:40.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fe0ab0700 1 Processor -- start 2024-02-20T13:43:40.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fe0ab0700 1 -- start start 2024-02-20T13:43:40.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fe0ab0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fe0ab0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc105be0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fda59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fda59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41200/0 (socket says 172.21.15.192:41200) 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.790+0000 7f5fda59c700 1 -- 172.21.15.192:0/712536446 learned_addr learned my addr 172.21.15.192:0/712536446 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fda59c700 1 -- 172.21.15.192:0/712536446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4005040 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fda59c700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5fd40089f0 tx=0x7f5fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fd4009320 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5fd4013ec0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fd4014d30 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc105de0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.791+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc106270 con 0x7f5fdc0f5480 2024-02-20T13:43:40.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.792+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f5fd40095f0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.792+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc045bb0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.792+0000 7f5fd2ffd700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 0x7f5fc80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:40.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.792+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5fd40423a0 con 0x7f5fdc0f5480 2024-02-20T13:43:40.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.792+0000 7f5fd9d9b700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 0x7f5fc80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:40.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.793+0000 7f5fd9d9b700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 0x7f5fc80335a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5fc40096a0 tx=0x7f5fc4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:40.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:40.797+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5fd400fe40 con 0x7f5fdc0f5480 2024-02-20T13:43:40.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.002+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fdc0faf70 con 0x7f5fdc0f5480 2024-02-20T13:43:40.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.003+0000 7f5fd2ffd700 1 -- 172.21.15.192:0/712536446 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5fd400fe40 con 0x7f5fdc0f5480 2024-02-20T13:43:40.992 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:40.992 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.004+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 msgr2=0x7f5fc80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.004+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 0x7f5fc80335a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5fc40096a0 tx=0x7f5fc4006b40).stop 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.004+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 msgr2=0x7f5fdc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.004+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5fd40089f0 tx=0x7f5fd400b040).stop 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 shutdown_connections 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5fc80310e0 0x7f5fc80335a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 --2- 172.21.15.192:0/712536446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fdc0f5480 0x7f5fdc1056f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:40.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 >> 172.21.15.192:0/712536446 conn(0x7f5fdc0f0be0 msgr2=0x7f5fdc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:40.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 shutdown_connections 2024-02-20T13:43:40.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:41.005+0000 7f5fe0ab0700 1 -- 172.21.15.192:0/712536446 wait complete. 2024-02-20T13:43:40.995 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:41.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:39.348099+0000 mgr.smithi158.gmoobf (mgr.14156) 84 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:41.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:41.004287+0000 mon.smithi158 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.192:0/712536446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:42.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:42.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:43.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:41.348429+0000 mgr.smithi158.gmoobf (mgr.14156) 85 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 -- 172.21.15.192:0/2419472663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 msgr2=0x7f44300f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 --2- 172.21.15.192:0/2419472663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44300f4c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4424009fb0 tx=0x7f4424009b30).stop 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 -- 172.21.15.192:0/2419472663 shutdown_connections 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 --2- 172.21.15.192:0/2419472663 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44300f4c60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 -- 172.21.15.192:0/2419472663 >> 172.21.15.192:0/2419472663 conn(0x7f44300f0360 msgr2=0x7f44300f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 -- 172.21.15.192:0/2419472663 shutdown_connections 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 -- 172.21.15.192:0/2419472663 wait complete. 2024-02-20T13:43:44.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.363+0000 7f44367be700 1 Processor -- start 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f44367be700 1 -- start start 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f44367be700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f44367be700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4430106cb0 con 0x7f44300f4840 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52234/0 (socket says 172.21.15.192:52234) 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442ffff700 1 -- 172.21.15.192:0/3723973339 learned_addr learned my addr 172.21.15.192:0/3723973339 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442ffff700 1 -- 172.21.15.192:0/3723973339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4424005040 con 0x7f44300f4840 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442ffff700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f44240089f0 tx=0x7f442400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.364+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4424009320 con 0x7f44300f4840 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430106eb0 con 0x7f44300f4840 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4424013ec0 con 0x7f44300f4840 2024-02-20T13:43:44.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4424014d30 con 0x7f44300f4840 2024-02-20T13:43:44.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4430107340 con 0x7f44300f4840 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f44240095f0 con 0x7f44300f4840 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.365+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4430045bb0 con 0x7f44300f4840 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.366+0000 7f442cff9700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 0x7f4418033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.366+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4424042490 con 0x7f44300f4840 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.366+0000 7f442f7fe700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 0x7f4418033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:44.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.366+0000 7f442f7fe700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 0x7f4418033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f44200096a0 tx=0x7f4420006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:44.359 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.370+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f442401d7f0 con 0x7f44300f4840 2024-02-20T13:43:44.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.575+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4430108190 con 0x7f44300f4840 2024-02-20T13:43:44.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.576+0000 7f442cff9700 1 -- 172.21.15.192:0/3723973339 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f442400fe40 con 0x7f44300f4840 2024-02-20T13:43:44.565 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:44.566 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 msgr2=0x7f4418033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 0x7f4418033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f44200096a0 tx=0x7f4420006b40).stop 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 msgr2=0x7f44301067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f44240089f0 tx=0x7f442400b040).stop 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 shutdown_connections 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4418031090 0x7f4418033550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 --2- 172.21.15.192:0/3723973339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44300f4840 0x7f44301067c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 >> 172.21.15.192:0/3723973339 conn(0x7f44300f0360 msgr2=0x7f44300f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 shutdown_connections 2024-02-20T13:43:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:44.578+0000 7f44367be700 1 -- 172.21.15.192:0/3723973339 wait complete. 2024-02-20T13:43:44.568 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:45.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:43.348670+0000 mgr.smithi158.gmoobf (mgr.14156) 86 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:45.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:44.577510+0000 mon.smithi158 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.192:0/3723973339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:46.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:46.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:45.348957+0000 mgr.smithi158.gmoobf (mgr.14156) 87 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 -- 172.21.15.192:0/2067704883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 msgr2=0x7fe9840f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 --2- 172.21.15.192:0/2067704883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9840f58a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe978009fb0 tx=0x7fe978009b30).stop 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 -- 172.21.15.192:0/2067704883 shutdown_connections 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 --2- 172.21.15.192:0/2067704883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9840f58a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 -- 172.21.15.192:0/2067704883 >> 172.21.15.192:0/2067704883 conn(0x7fe9840f0be0 msgr2=0x7fe9840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 -- 172.21.15.192:0/2067704883 shutdown_connections 2024-02-20T13:43:47.880 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.890+0000 7fe98951a700 1 -- 172.21.15.192:0/2067704883 wait complete. 2024-02-20T13:43:47.882 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.891+0000 7fe98951a700 1 Processor -- start 2024-02-20T13:43:47.882 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.891+0000 7fe98951a700 1 -- start start 2024-02-20T13:43:47.882 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.891+0000 7fe98951a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:47.882 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.891+0000 7fe98951a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984105be0 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe982ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe982ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52252/0 (socket says 172.21.15.192:52252) 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe982ffd700 1 -- 172.21.15.192:0/3653429910 learned_addr learned my addr 172.21.15.192:0/3653429910 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe982ffd700 1 -- 172.21.15.192:0/3653429910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978005040 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe982ffd700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe9780089f0 tx=0x7fe97800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe978009320 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe984105de0 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe978013ec0 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe978014d30 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.892+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984106270 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.893+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe978014420 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.893+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe984045bb0 con 0x7fe9840f5480 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.893+0000 7fe96b7fe700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 0x7fe96c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:47.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.893+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe978042450 con 0x7fe9840f5480 2024-02-20T13:43:47.884 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.894+0000 7fe9827fc700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 0x7fe96c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:47.884 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.894+0000 7fe9827fc700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 0x7fe96c033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe9740096a0 tx=0x7fe974006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:47.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:47.898+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe97801d7f0 con 0x7fe9840f5480 2024-02-20T13:43:48.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.102+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9840faf70 con 0x7fe9840f5480 2024-02-20T13:43:48.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.103+0000 7fe96b7fe700 1 -- 172.21.15.192:0/3653429910 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe97801d7f0 con 0x7fe9840f5480 2024-02-20T13:43:48.093 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:48.093 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:48.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 msgr2=0x7fe96c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:48.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 0x7fe96c033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe9740096a0 tx=0x7fe974006b40).stop 2024-02-20T13:43:48.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 msgr2=0x7fe9841056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe9780089f0 tx=0x7fe97800b040).stop 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 shutdown_connections 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe96c031090 0x7fe96c033550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 --2- 172.21.15.192:0/3653429910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9840f5480 0x7fe9841056f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 >> 172.21.15.192:0/3653429910 conn(0x7fe9840f0be0 msgr2=0x7fe9840f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 shutdown_connections 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:48.105+0000 7fe98951a700 1 -- 172.21.15.192:0/3653429910 wait complete. 2024-02-20T13:43:48.095 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:48.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:48.104618+0000 mon.smithi158 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.192:0/3653429910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:49.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:47.349223+0000 mgr.smithi158.gmoobf (mgr.14156) 88 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:49.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:49.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 -- 172.21.15.192:0/3747468704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 msgr2=0x7f47ac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 --2- 172.21.15.192:0/3747468704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac0f69b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f47a8004760 tx=0x7f47a8009ad0).stop 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 -- 172.21.15.192:0/3747468704 shutdown_connections 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 --2- 172.21.15.192:0/3747468704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac0f69b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 -- 172.21.15.192:0/3747468704 >> 172.21.15.192:0/3747468704 conn(0x7f47ac0f1cd0 msgr2=0x7f47ac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 -- 172.21.15.192:0/3747468704 shutdown_connections 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.426+0000 7f47b39e2700 1 -- 172.21.15.192:0/3747468704 wait complete. 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b39e2700 1 Processor -- start 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b39e2700 1 -- start start 2024-02-20T13:43:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b39e2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:51.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b39e2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac069f00 con 0x7f47ac0f6590 2024-02-20T13:43:51.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b177e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:51.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b177e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52262/0 (socket says 172.21.15.192:52262) 2024-02-20T13:43:51.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b177e700 1 -- 172.21.15.192:0/2507495908 learned_addr learned my addr 172.21.15.192:0/2507495908 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:51.418 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b177e700 1 -- 172.21.15.192:0/2507495908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009d70 con 0x7f47ac0f6590 2024-02-20T13:43:51.418 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.427+0000 7f47b177e700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f47a8004170 tx=0x7f47a8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.428+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a800e040 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.428+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f47a8008ef0 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.428+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac06a0a0 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.428+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a8014c90 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.428+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac06a530 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f47a8014420 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac0591e0 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47a1ffb700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 0x7f4798033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47a8042210 con 0x7f47ac0f6590 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47b0f7d700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 0x7f4798033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.429+0000 7f47b0f7d700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 0x7f4798033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:51.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.434+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f47a8026300 con 0x7f47ac0f6590 2024-02-20T13:43:51.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:49.349533+0000 mgr.smithi158.gmoobf (mgr.14156) 89 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:51.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.639+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47ac045bb0 con 0x7f47ac0f6590 2024-02-20T13:43:51.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.640+0000 7f47a1ffb700 1 -- 172.21.15.192:0/2507495908 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f47a8012b20 con 0x7f47ac0f6590 2024-02-20T13:43:51.628 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:51.629 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:51.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 msgr2=0x7f4798033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:51.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 0x7f4798033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).stop 2024-02-20T13:43:51.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 msgr2=0x7f47ac06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f47a8004170 tx=0x7f47a8009ad0).stop 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 shutdown_connections 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4798031090 0x7f4798033550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 --2- 172.21.15.192:0/2507495908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47ac0f6590 0x7f47ac06ba80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 >> 172.21.15.192:0/2507495908 conn(0x7f47ac0f1cd0 msgr2=0x7f47ac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 shutdown_connections 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:51.641+0000 7f47b39e2700 1 -- 172.21.15.192:0/2507495908 wait complete. 2024-02-20T13:43:51.631 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:52.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:52 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:51.640994+0000 mon.smithi158 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.192:0/2507495908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:53.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:53.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:53.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:51.349817+0000 mgr.smithi158.gmoobf (mgr.14156) 90 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 -- 172.21.15.192:0/1506142187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0f5480 msgr2=0x7f111c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 --2- 172.21.15.192:0/1506142187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0f5480 0x7f111c0f58a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f110c009fb0 tx=0x7f110c009b30).stop 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 -- 172.21.15.192:0/1506142187 shutdown_connections 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 --2- 172.21.15.192:0/1506142187 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0f5480 0x7f111c0f58a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 -- 172.21.15.192:0/1506142187 >> 172.21.15.192:0/1506142187 conn(0x7f111c0f0be0 msgr2=0x7f111c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 -- 172.21.15.192:0/1506142187 shutdown_connections 2024-02-20T13:43:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.151+0000 7f1123b97700 1 -- 172.21.15.192:0/1506142187 wait complete. 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1123b97700 1 Processor -- start 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1123b97700 1 -- start start 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1123b97700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1123b97700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c109af0 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1121933700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1121933700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53002/0 (socket says 172.21.15.192:53002) 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1121933700 1 -- 172.21.15.192:0/4179078462 learned_addr learned my addr 172.21.15.192:0/4179078462 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.152+0000 7f1121933700 1 -- 172.21.15.192:0/4179078462 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c005040 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f1121933700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f110c0134d0 tx=0x7f110c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110c013ee0 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c109c30 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f110c013680 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110c01ee90 con 0x7f111c0602e0 2024-02-20T13:43:55.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c10a060 con 0x7f111c0602e0 2024-02-20T13:43:55.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.153+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f110c025060 con 0x7f111c0602e0 2024-02-20T13:43:55.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.154+0000 7f11127fc700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 0x7f1108033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:55.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.154+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c045bb0 con 0x7f111c0602e0 2024-02-20T13:43:55.145 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.154+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f110c042350 con 0x7f111c0602e0 2024-02-20T13:43:55.145 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.154+0000 7f1121132700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 0x7f1108033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:55.145 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.154+0000 7f1121132700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 0x7f1108033500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f11180096a0 tx=0x7f1118006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:55.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.158+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f110c01c520 con 0x7f111c0602e0 2024-02-20T13:43:55.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.365+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f111c0fb060 con 0x7f111c0602e0 2024-02-20T13:43:55.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.366+0000 7f11127fc700 1 -- 172.21.15.192:0/4179078462 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f110c01c340 con 0x7f111c0602e0 2024-02-20T13:43:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:55.355 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:55.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 msgr2=0x7f1108033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:55.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 0x7f1108033500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f11180096a0 tx=0x7f1118006b40).stop 2024-02-20T13:43:55.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 msgr2=0x7f111c109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f110c0134d0 tx=0x7f110c009b30).stop 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 shutdown_connections 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1108031040 0x7f1108033500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 --2- 172.21.15.192:0/4179078462 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f111c0602e0 0x7f111c109600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 >> 172.21.15.192:0/4179078462 conn(0x7f111c0f0be0 msgr2=0x7f111c0f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 shutdown_connections 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:55.367+0000 7f1123b97700 1 -- 172.21.15.192:0/4179078462 wait complete. 2024-02-20T13:43:55.357 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:55.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:53.350161+0000 mgr.smithi158.gmoobf (mgr.14156) 91 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:55.367006+0000 mon.smithi158 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.192:0/4179078462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:43:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:55.460758+0000 mon.smithi158 (mon.0) 152 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:43:56.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:55.461490+0000 mon.smithi158 (mon.0) 153 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:43:56.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:43:56.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:43:57.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:55.350618+0000 mgr.smithi158.gmoobf (mgr.14156) 92 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:58.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 -- 172.21.15.192:0/880412421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 msgr2=0x7faf040f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 --2- 172.21.15.192:0/880412421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf040f69a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7faef8009fb0 tx=0x7faef8009b30).stop 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 -- 172.21.15.192:0/880412421 shutdown_connections 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 --2- 172.21.15.192:0/880412421 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf040f69a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 -- 172.21.15.192:0/880412421 >> 172.21.15.192:0/880412421 conn(0x7faf040f1ce0 msgr2=0x7faf040f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 -- 172.21.15.192:0/880412421 shutdown_connections 2024-02-20T13:43:58.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.917+0000 7faf0b25d700 1 -- 172.21.15.192:0/880412421 wait complete. 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf0b25d700 1 Processor -- start 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf0b25d700 1 -- start start 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf0b25d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf0b25d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04069f00 con 0x7faf040f6580 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf08ff9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf08ff9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53024/0 (socket says 172.21.15.192:53024) 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.918+0000 7faf08ff9700 1 -- 172.21.15.192:0/3651010846 learned_addr learned my addr 172.21.15.192:0/3651010846 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:43:58.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf08ff9700 1 -- 172.21.15.192:0/3651010846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8005040 con 0x7faf040f6580 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf08ff9700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7faef80089f0 tx=0x7faef800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef8009320 con 0x7faf040f6580 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0406a0a0 con 0x7faf040f6580 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faef8013ec0 con 0x7faf040f6580 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef8014d30 con 0x7faf040f6580 2024-02-20T13:43:58.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.919+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0406a530 con 0x7faf040f6580 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.920+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7faef80095f0 con 0x7faf040f6580 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.920+0000 7faf017fa700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 0x7faef40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.920+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf040591e0 con 0x7faf040f6580 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.920+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faef8042490 con 0x7faf040f6580 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.921+0000 7faf03fff700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 0x7faef40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:43:58.913 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.921+0000 7faf03fff700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 0x7faef40335a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7faef00096a0 tx=0x7faef0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:43:58.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:58.925+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faef801d7f0 con 0x7faf040f6580 2024-02-20T13:43:59.119 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.130+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf04045bb0 con 0x7faf040f6580 2024-02-20T13:43:59.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.131+0000 7faf017fa700 1 -- 172.21.15.192:0/3651010846 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faef800fe40 con 0x7faf040f6580 2024-02-20T13:43:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:43:59.120 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:43:59.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 msgr2=0x7faef40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:59.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 0x7faef40335a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7faef00096a0 tx=0x7faef0006b40).stop 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 msgr2=0x7faf0406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7faef80089f0 tx=0x7faef800b040).stop 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 shutdown_connections 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faef40310e0 0x7faef40335a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 --2- 172.21.15.192:0/3651010846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf040f6580 0x7faf0406ba80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 >> 172.21.15.192:0/3651010846 conn(0x7faf040f1ce0 msgr2=0x7faf040f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 shutdown_connections 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:43:59.132+0000 7faf0b25d700 1 -- 172.21.15.192:0/3651010846 wait complete. 2024-02-20T13:43:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:43:59.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:57.351046+0000 mgr.smithi158.gmoobf (mgr.14156) 93 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:43:59.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:43:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:43:59.132171+0000 mon.smithi158 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.192:0/3651010846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:00.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:00.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:01.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:43:59.351396+0000 mgr.smithi158.gmoobf (mgr.14156) 94 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:02.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 -- 172.21.15.192:0/2948364232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 msgr2=0x7fe8c00f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2948364232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c00f8fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe8b0009fb0 tx=0x7fe8b0009b30).stop 2024-02-20T13:44:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 -- 172.21.15.192:0/2948364232 shutdown_connections 2024-02-20T13:44:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2948364232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c00f8fb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 -- 172.21.15.192:0/2948364232 >> 172.21.15.192:0/2948364232 conn(0x7fe8c00f0a30 msgr2=0x7fe8c00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.547+0000 7fe8bffff700 1 -- 172.21.15.192:0/2948364232 shutdown_connections 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.548+0000 7fe8bffff700 1 -- 172.21.15.192:0/2948364232 wait complete. 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.548+0000 7fe8bffff700 1 Processor -- start 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.548+0000 7fe8bffff700 1 -- start start 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.548+0000 7fe8bffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.548+0000 7fe8bffff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c0106aa0 con 0x7fe8c0060360 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8beffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8beffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53724/0 (socket says 172.21.15.192:53724) 2024-02-20T13:44:02.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8beffd700 1 -- 172.21.15.192:0/2007982605 learned_addr learned my addr 172.21.15.192:0/2007982605 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:02.542 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8beffd700 1 -- 172.21.15.192:0/2007982605 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0005040 con 0x7fe8c0060360 2024-02-20T13:44:02.542 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8beffd700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe8b0009f80 tx=0x7fe8b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:02.542 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0009320 con 0x7fe8c0060360 2024-02-20T13:44:02.542 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c0106ca0 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe8b0013680 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b001eda0 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.549+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c0107130 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.550+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe8b001e420 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.550+0000 7fe8a77fe700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 0x7fe8a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.550+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8b0042340 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.550+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c0045bb0 con 0x7fe8c0060360 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.550+0000 7fe8be7fc700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 0x7fe8a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:02.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.551+0000 7fe8be7fc700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 0x7fe8a8033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe8b40096a0 tx=0x7fe8b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:02.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.555+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe8b001e700 con 0x7fe8c0060360 2024-02-20T13:44:02.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.762+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8c00fbfa0 con 0x7fe8c0060360 2024-02-20T13:44:02.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.763+0000 7fe8a77fe700 1 -- 172.21.15.192:0/2007982605 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8b001cb10 con 0x7fe8c0060360 2024-02-20T13:44:02.752 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:02.752 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 msgr2=0x7fe8a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 0x7fe8a8033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe8b40096a0 tx=0x7fe8b4006b40).stop 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 msgr2=0x7fe8c01065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe8b0009f80 tx=0x7fe8b000b040).stop 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 shutdown_connections 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe8a8031090 0x7fe8a8033550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 --2- 172.21.15.192:0/2007982605 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8c0060360 0x7fe8c01065b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 >> 172.21.15.192:0/2007982605 conn(0x7fe8c00f0a30 msgr2=0x7fe8c00f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 shutdown_connections 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:02.764+0000 7fe8bffff700 1 -- 172.21.15.192:0/2007982605 wait complete. 2024-02-20T13:44:02.754 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:03.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:01.351771+0000 mgr.smithi158.gmoobf (mgr.14156) 95 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:03.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:02.764027+0000 mon.smithi158 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.192:0/2007982605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:04.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:04.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:05.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:03.352121+0000 mgr.smithi158.gmoobf (mgr.14156) 96 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- 172.21.15.192:0/16878716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 msgr2=0x7f9a140f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/16878716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a140f56f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9a04009fb0 tx=0x7f9a04009b30).stop 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- 172.21.15.192:0/16878716 shutdown_connections 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/16878716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a140f56f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- 172.21.15.192:0/16878716 >> 172.21.15.192:0/16878716 conn(0x7f9a140f0a30 msgr2=0x7f9a140f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- 172.21.15.192:0/16878716 shutdown_connections 2024-02-20T13:44:06.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- 172.21.15.192:0/16878716 wait complete. 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 Processor -- start 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.256+0000 7f9a18eb5700 1 -- start start 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a18eb5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a18eb5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14106b30 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53748/0 (socket says 172.21.15.192:53748) 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a137fe700 1 -- 172.21.15.192:0/2378953471 learned_addr learned my addr 172.21.15.192:0/2378953471 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a137fe700 1 -- 172.21.15.192:0/2378953471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04005040 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a137fe700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9a040089f0 tx=0x7f9a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04009320 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a14106d30 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a04013ec0 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04014d30 con 0x7f9a140f52d0 2024-02-20T13:44:06.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.257+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a141071c0 con 0x7f9a140f52d0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.258+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9a040095f0 con 0x7f9a140f52d0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.258+0000 7f99fbfff700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 0x7f99fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.258+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a14045bb0 con 0x7f9a140f52d0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.258+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a04042480 con 0x7f9a140f52d0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.258+0000 7f9a12ffd700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 0x7f99fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.259+0000 7f9a12ffd700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 0x7f99fc033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9a080096a0 tx=0x7f9a08006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:06.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.264+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a0401d980 con 0x7f9a140f52d0 2024-02-20T13:44:06.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.470+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a14108040 con 0x7f9a140f52d0 2024-02-20T13:44:06.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f99fbfff700 1 -- 172.21.15.192:0/2378953471 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9a0401d380 con 0x7f9a140f52d0 2024-02-20T13:44:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:06.460 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:06.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 msgr2=0x7f99fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:06.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 0x7f99fc033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9a080096a0 tx=0x7f9a08006b40).stop 2024-02-20T13:44:06.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 msgr2=0x7f9a14106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:06.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9a040089f0 tx=0x7f9a0400b040).stop 2024-02-20T13:44:06.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 shutdown_connections 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f99fc031090 0x7f99fc033550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 --2- 172.21.15.192:0/2378953471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a140f52d0 0x7f9a14106640 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 >> 172.21.15.192:0/2378953471 conn(0x7f9a140f0a30 msgr2=0x7f9a140f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 shutdown_connections 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:06.472+0000 7f9a18eb5700 1 -- 172.21.15.192:0/2378953471 wait complete. 2024-02-20T13:44:06.462 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:07.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:05.352435+0000 mgr.smithi158.gmoobf (mgr.14156) 97 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:07.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:06.472197+0000 mon.smithi158 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.192:0/2378953471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:08.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:08.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:09.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:07.352762+0000 mgr.smithi158.gmoobf (mgr.14156) 98 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 -- 172.21.15.192:0/802434924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 msgr2=0x7f9d500f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/802434924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d500f56f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9d44009fb0 tx=0x7f9d44009b30).stop 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 -- 172.21.15.192:0/802434924 shutdown_connections 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/802434924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d500f56f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 -- 172.21.15.192:0/802434924 >> 172.21.15.192:0/802434924 conn(0x7f9d500f0a30 msgr2=0x7f9d500f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 -- 172.21.15.192:0/802434924 shutdown_connections 2024-02-20T13:44:09.885 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.895+0000 7f9d54a5e700 1 -- 172.21.15.192:0/802434924 wait complete. 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.896+0000 7f9d54a5e700 1 Processor -- start 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.896+0000 7f9d54a5e700 1 -- start start 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.896+0000 7f9d54a5e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.896+0000 7f9d54a5e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50060320 con 0x7f9d500f52d0 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d4ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d4ed9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53758/0 (socket says 172.21.15.192:53758) 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d4ed9d700 1 -- 172.21.15.192:0/3770841854 learned_addr learned my addr 172.21.15.192:0/3770841854 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d4ed9d700 1 -- 172.21.15.192:0/3770841854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44005040 con 0x7f9d500f52d0 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d4ed9d700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9d440089f0 tx=0x7f9d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d44009320 con 0x7f9d500f52d0 2024-02-20T13:44:09.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d50060520 con 0x7f9d500f52d0 2024-02-20T13:44:09.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9d44013ec0 con 0x7f9d500f52d0 2024-02-20T13:44:09.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d44014d30 con 0x7f9d500f52d0 2024-02-20T13:44:09.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.897+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d501099c0 con 0x7f9d500f52d0 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.898+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9d440095f0 con 0x7f9d500f52d0 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.898+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d50045bb0 con 0x7f9d500f52d0 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.898+0000 7f9d377fe700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 0x7f9d38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.898+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9d440423d0 con 0x7f9d500f52d0 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.899+0000 7f9d4e59c700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 0x7f9d38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:09.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.899+0000 7f9d4e59c700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 0x7f9d38033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9d400096a0 tx=0x7f9d40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:09.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:09.903+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9d4401d7f0 con 0x7f9d500f52d0 2024-02-20T13:44:10.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.110+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d5010a1d0 con 0x7f9d500f52d0 2024-02-20T13:44:10.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.111+0000 7f9d377fe700 1 -- 172.21.15.192:0/3770841854 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d4400fe40 con 0x7f9d500f52d0 2024-02-20T13:44:10.099 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:10.099 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:10.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.112+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 msgr2=0x7f9d38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.112+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 0x7f9d38033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9d400096a0 tx=0x7f9d40006b40).stop 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 msgr2=0x7f9d50109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9d440089f0 tx=0x7f9d4400b040).stop 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 shutdown_connections 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9d38031090 0x7f9d38033550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 --2- 172.21.15.192:0/3770841854 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d500f52d0 0x7f9d50109310 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 >> 172.21.15.192:0/3770841854 conn(0x7f9d500f0a30 msgr2=0x7f9d500f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 shutdown_connections 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:10.113+0000 7f9d54a5e700 1 -- 172.21.15.192:0/3770841854 wait complete. 2024-02-20T13:44:10.102 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:10.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:10.111226+0000 mon.smithi158 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.192:0/3770841854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:11.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:09.353061+0000 mgr.smithi158.gmoobf (mgr.14156) 99 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:11.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:11.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:13.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:11.353314+0000 mgr.smithi158.gmoobf (mgr.14156) 100 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:13.578 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- 172.21.15.192:0/1413471708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 msgr2=0x7f64a80f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:13.578 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/1413471708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a80f6800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6498009fb0 tx=0x7f6498009b30).stop 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- 172.21.15.192:0/1413471708 shutdown_connections 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/1413471708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a80f6800 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- 172.21.15.192:0/1413471708 >> 172.21.15.192:0/1413471708 conn(0x7f64a80f1b40 msgr2=0x7f64a80f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- 172.21.15.192:0/1413471708 shutdown_connections 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- 172.21.15.192:0/1413471708 wait complete. 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 Processor -- start 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.589+0000 7f64ad2d1700 1 -- start start 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64ad2d1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64ad2d1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8106b10 con 0x7f64a80f63e0 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41550/0 (socket says 172.21.15.192:41550) 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a7fff700 1 -- 172.21.15.192:0/150512746 learned_addr learned my addr 172.21.15.192:0/150512746 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a7fff700 1 -- 172.21.15.192:0/150512746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498005040 con 0x7f64a80f63e0 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a7fff700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f64980089f0 tx=0x7f649800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:13.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498009320 con 0x7f64a80f63e0 2024-02-20T13:44:13.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a8106d10 con 0x7f64a80f63e0 2024-02-20T13:44:13.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6498013ec0 con 0x7f64a80f63e0 2024-02-20T13:44:13.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498014d30 con 0x7f64a80f63e0 2024-02-20T13:44:13.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.590+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a81071a0 con 0x7f64a80f63e0 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.591+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6498014420 con 0x7f64a80f63e0 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.591+0000 7f64a4ff9700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 0x7f6490033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.591+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a8045bb0 con 0x7f64a80f63e0 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.591+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6498042440 con 0x7f64a80f63e0 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.592+0000 7f64a77fe700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 0x7f6490033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:13.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.592+0000 7f64a77fe700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 0x7f6490033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f649c0096a0 tx=0x7f649c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:13.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.596+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6498014750 con 0x7f64a80f63e0 2024-02-20T13:44:13.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.805+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64a8107fa0 con 0x7f64a80f63e0 2024-02-20T13:44:13.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.805+0000 7f64a4ff9700 1 -- 172.21.15.192:0/150512746 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f649802b020 con 0x7f64a80f63e0 2024-02-20T13:44:13.794 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:13.794 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:13.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 msgr2=0x7f6490033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:13.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 0x7f6490033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f649c0096a0 tx=0x7f649c006b40).stop 2024-02-20T13:44:13.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 msgr2=0x7f64a8106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:13.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f64980089f0 tx=0x7f649800b040).stop 2024-02-20T13:44:13.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 shutdown_connections 2024-02-20T13:44:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6490031090 0x7f6490033550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 --2- 172.21.15.192:0/150512746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64a80f63e0 0x7f64a8106620 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 >> 172.21.15.192:0/150512746 conn(0x7f64a80f1b40 msgr2=0x7f64a80f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 shutdown_connections 2024-02-20T13:44:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:13.807+0000 7f64ad2d1700 1 -- 172.21.15.192:0/150512746 wait complete. 2024-02-20T13:44:13.797 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:14.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:14 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:13.805843+0000 mon.smithi158 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.192:0/150512746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:15.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:15.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:15.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:13.353646+0000 mgr.smithi158.gmoobf (mgr.14156) 101 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:17.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- 172.21.15.192:0/782423030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 msgr2=0x7fe4c40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:17.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/782423030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c40f9190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe4b4009fb0 tx=0x7fe4b4009b30).stop 2024-02-20T13:44:17.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- 172.21.15.192:0/782423030 shutdown_connections 2024-02-20T13:44:17.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/782423030 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c40f9190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:17.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- 172.21.15.192:0/782423030 >> 172.21.15.192:0/782423030 conn(0x7fe4c40f0c10 msgr2=0x7fe4c40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- 172.21.15.192:0/782423030 shutdown_connections 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- 172.21.15.192:0/782423030 wait complete. 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 Processor -- start 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- start start 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.394+0000 7fe4cbf30700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c4106c60 con 0x7fe4c4060360 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4c9ccc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4c9ccc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41558/0 (socket says 172.21.15.192:41558) 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4c9ccc700 1 -- 172.21.15.192:0/353805465 learned_addr learned my addr 172.21.15.192:0/353805465 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4c9ccc700 1 -- 172.21.15.192:0/353805465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b4005040 con 0x7fe4c4060360 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4c9ccc700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe4b4009f80 tx=0x7fe4b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b4009320 con 0x7fe4c4060360 2024-02-20T13:44:17.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c4106e60 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe4b4013680 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b401eda0 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.395+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c41072f0 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.396+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe4b401e420 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.396+0000 7fe4ba7fc700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 0x7fe4b0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.396+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c4045bb0 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.396+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe4b4042420 con 0x7fe4c4060360 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.396+0000 7fe4c94cb700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 0x7fe4b0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:17.386 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.397+0000 7fe4c94cb700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 0x7fe4b0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe4c00096a0 tx=0x7fe4c0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:17.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.401+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe4b401d380 con 0x7fe4c4060360 2024-02-20T13:44:17.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:15.353995+0000 mgr.smithi158.gmoobf (mgr.14156) 102 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:17.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.610+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4c40fc090 con 0x7fe4c4060360 2024-02-20T13:44:17.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.610+0000 7fe4ba7fc700 1 -- 172.21.15.192:0/353805465 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4b401cad0 con 0x7fe4c4060360 2024-02-20T13:44:17.598 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:17.598 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 msgr2=0x7fe4b0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 0x7fe4b0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe4c00096a0 tx=0x7fe4c0006b40).stop 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 msgr2=0x7fe4c4106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe4b4009f80 tx=0x7fe4b400b040).stop 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 shutdown_connections 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4b0031090 0x7fe4b0033550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 --2- 172.21.15.192:0/353805465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4c4060360 0x7fe4c4106770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 >> 172.21.15.192:0/353805465 conn(0x7fe4c40f0c10 msgr2=0x7fe4c40f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:17.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 shutdown_connections 2024-02-20T13:44:17.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:17.612+0000 7fe4cbf30700 1 -- 172.21.15.192:0/353805465 wait complete. 2024-02-20T13:44:17.601 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:18.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:17.610822+0000 mon.smithi158 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.192:0/353805465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:19.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:19.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:19.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:17.354411+0000 mgr.smithi158.gmoobf (mgr.14156) 103 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:21.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 -- 172.21.15.192:0/3793117609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 msgr2=0x7f26f80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:21.110 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 --2- 172.21.15.192:0/3793117609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f80f58a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f26e8009fb0 tx=0x7f26e8009b30).stop 2024-02-20T13:44:21.110 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 -- 172.21.15.192:0/3793117609 shutdown_connections 2024-02-20T13:44:21.113 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 --2- 172.21.15.192:0/3793117609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f80f58a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 -- 172.21.15.192:0/3793117609 >> 172.21.15.192:0/3793117609 conn(0x7f26f80f0be0 msgr2=0x7f26f80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 -- 172.21.15.192:0/3793117609 shutdown_connections 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.114+0000 7f27003ab700 1 -- 172.21.15.192:0/3793117609 wait complete. 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.115+0000 7f27003ab700 1 Processor -- start 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.115+0000 7f27003ab700 1 -- start start 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.115+0000 7f27003ab700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.115+0000 7f27003ab700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8105be0 con 0x7f26f80f5480 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26fe147700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26fe147700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41582/0 (socket says 172.21.15.192:41582) 2024-02-20T13:44:21.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26fe147700 1 -- 172.21.15.192:0/3148522424 learned_addr learned my addr 172.21.15.192:0/3148522424 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26fe147700 1 -- 172.21.15.192:0/3148522424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8005040 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26fe147700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f26e80089f0 tx=0x7f26e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e8009320 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f26e8013ec0 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f8105de0 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e8014d30 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.116+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f8106270 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.117+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f26e80095f0 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.117+0000 7f26eeffd700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 0x7f26e40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.117+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8045bb0 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.117+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26e80423d0 con 0x7f26f80f5480 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.117+0000 7f26fd946700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 0x7f26e40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:21.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.118+0000 7f26fd946700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 0x7f26e40335a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f26f40096a0 tx=0x7f26f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:21.116 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.122+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f26e800fe40 con 0x7f26f80f5480 2024-02-20T13:44:21.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.328+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26f80fb060 con 0x7f26f80f5480 2024-02-20T13:44:21.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.329+0000 7f26eeffd700 1 -- 172.21.15.192:0/3148522424 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26e800fe40 con 0x7f26f80f5480 2024-02-20T13:44:21.318 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:21.318 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:21.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 msgr2=0x7f26e40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:21.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 0x7f26e40335a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f26f40096a0 tx=0x7f26f4006b40).stop 2024-02-20T13:44:21.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 msgr2=0x7f26f81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:21.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f26e80089f0 tx=0x7f26e800b040).stop 2024-02-20T13:44:21.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 shutdown_connections 2024-02-20T13:44:21.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f26e40310e0 0x7f26e40335a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:21.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 --2- 172.21.15.192:0/3148522424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26f80f5480 0x7f26f81056f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:21.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 >> 172.21.15.192:0/3148522424 conn(0x7f26f80f0be0 msgr2=0x7f26f80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:21.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 shutdown_connections 2024-02-20T13:44:21.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:21.330+0000 7f27003ab700 1 -- 172.21.15.192:0/3148522424 wait complete. 2024-02-20T13:44:21.322 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:21.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:19.354822+0000 mgr.smithi158.gmoobf (mgr.14156) 104 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:22.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:21.330193+0000 mon.smithi158 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.192:0/3148522424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:22.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:22.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:21.355236+0000 mgr.smithi158.gmoobf (mgr.14156) 105 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:24.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 -- 172.21.15.192:0/2591287531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 msgr2=0x7fa8d00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 --2- 172.21.15.192:0/2591287531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d00f58a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa8c4009fb0 tx=0x7fa8c4009b30).stop 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 -- 172.21.15.192:0/2591287531 shutdown_connections 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 --2- 172.21.15.192:0/2591287531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d00f58a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 -- 172.21.15.192:0/2591287531 >> 172.21.15.192:0/2591287531 conn(0x7fa8d00f0be0 msgr2=0x7fa8d00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 -- 172.21.15.192:0/2591287531 shutdown_connections 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.753+0000 7fa8d7895700 1 -- 172.21.15.192:0/2591287531 wait complete. 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.754+0000 7fa8d7895700 1 Processor -- start 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.754+0000 7fa8d7895700 1 -- start start 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.754+0000 7fa8d7895700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:24.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.754+0000 7fa8d7895700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00602b0 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d5631700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d5631700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:33456/0 (socket says 172.21.15.192:33456) 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d5631700 1 -- 172.21.15.192:0/536906362 learned_addr learned my addr 172.21.15.192:0/536906362 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d5631700 1 -- 172.21.15.192:0/536906362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4005040 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d5631700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa8c40089f0 tx=0x7fa8c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c4009320 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d00604b0 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa8c4013ec0 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.755+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c4014d30 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.756+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d0109ba0 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.756+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa8c40095f0 con 0x7fa8d00f5480 2024-02-20T13:44:24.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.756+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d0045bb0 con 0x7fa8d00f5480 2024-02-20T13:44:24.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.756+0000 7fa8c1ffb700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 0x7fa8bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:24.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.757+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa8c4042490 con 0x7fa8d00f5480 2024-02-20T13:44:24.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.757+0000 7fa8d4e30700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 0x7fa8bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:24.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.757+0000 7fa8d4e30700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 0x7fa8bc033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa8cc0096a0 tx=0x7fa8cc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:24.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.761+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa8c401d7f0 con 0x7fa8d00f5480 2024-02-20T13:44:24.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.968+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8d010a350 con 0x7fa8d00f5480 2024-02-20T13:44:24.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.968+0000 7fa8c1ffb700 1 -- 172.21.15.192:0/536906362 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa8c400fe40 con 0x7fa8d00f5480 2024-02-20T13:44:24.957 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:24.957 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 msgr2=0x7fa8bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 0x7fa8bc033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa8cc0096a0 tx=0x7fa8cc006b40).stop 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 msgr2=0x7fa8d01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa8c40089f0 tx=0x7fa8c400b040).stop 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 shutdown_connections 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa8bc031090 0x7fa8bc033550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 --2- 172.21.15.192:0/536906362 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8d00f5480 0x7fa8d01094f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:24.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 >> 172.21.15.192:0/536906362 conn(0x7fa8d00f0be0 msgr2=0x7fa8d00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:24.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 shutdown_connections 2024-02-20T13:44:24.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:24.970+0000 7fa8d7895700 1 -- 172.21.15.192:0/536906362 wait complete. 2024-02-20T13:44:24.960 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:25.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:23.355618+0000 mgr.smithi158.gmoobf (mgr.14156) 106 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:25.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:24.969702+0000 mon.smithi158 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.192:0/536906362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:26.629 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:26.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:25.355971+0000 mgr.smithi158.gmoobf (mgr.14156) 107 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 -- 172.21.15.192:0/3286933963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 msgr2=0x7f13f80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 --2- 172.21.15.192:0/3286933963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f80f69b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f13ec004760 tx=0x7f13ec009ad0).stop 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 -- 172.21.15.192:0/3286933963 shutdown_connections 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 --2- 172.21.15.192:0/3286933963 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f80f69b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 -- 172.21.15.192:0/3286933963 >> 172.21.15.192:0/3286933963 conn(0x7f13f80f1cd0 msgr2=0x7f13f80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 -- 172.21.15.192:0/3286933963 shutdown_connections 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.462+0000 7f13fce8e700 1 -- 172.21.15.192:0/3286933963 wait complete. 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13fce8e700 1 Processor -- start 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13fce8e700 1 -- start start 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13fce8e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13fce8e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8069f00 con 0x7f13f80f6590 2024-02-20T13:44:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13f659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13f659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:33464/0 (socket says 172.21.15.192:33464) 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13f659c700 1 -- 172.21.15.192:0/2216707101 learned_addr learned my addr 172.21.15.192:0/2216707101 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13f659c700 1 -- 172.21.15.192:0/2216707101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec009d70 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.463+0000 7f13f659c700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f13ec004170 tx=0x7f13ec009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ec00e040 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f806a0a0 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f13ec008ef0 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ec014c90 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f806a530 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.464+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f13ec014420 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.465+0000 7f13e6ffd700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 0x7f13e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.465+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13ec042290 con 0x7f13f80f6590 2024-02-20T13:44:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.465+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f80591e0 con 0x7f13f80f6590 2024-02-20T13:44:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.465+0000 7f13f5d9b700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 0x7f13e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.465+0000 7f13f5d9b700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 0x7f13e0033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f13e80096a0 tx=0x7f13e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.469+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13ec026300 con 0x7f13f80f6590 2024-02-20T13:44:28.666 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.678+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13f8045bb0 con 0x7f13f80f6590 2024-02-20T13:44:28.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.678+0000 7f13e6ffd700 1 -- 172.21.15.192:0/2216707101 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f13f8045bb0 con 0x7f13f80f6590 2024-02-20T13:44:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:28.667 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:28.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 msgr2=0x7f13e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:28.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 0x7f13e0033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f13e80096a0 tx=0x7f13e8006b40).stop 2024-02-20T13:44:28.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 msgr2=0x7f13f806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f13ec004170 tx=0x7f13ec009ad0).stop 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 shutdown_connections 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f13e0031090 0x7f13e0033550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 --2- 172.21.15.192:0/2216707101 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13f80f6590 0x7f13f806ba80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 >> 172.21.15.192:0/2216707101 conn(0x7f13f80f1cd0 msgr2=0x7f13f80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 shutdown_connections 2024-02-20T13:44:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:28.680+0000 7f13fce8e700 1 -- 172.21.15.192:0/2216707101 wait complete. 2024-02-20T13:44:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:29.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:27.356269+0000 mgr.smithi158.gmoobf (mgr.14156) 108 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:29.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:28.679710+0000 mon.smithi158 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.192:0/2216707101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:30.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:30.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:31.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:29.356603+0000 mgr.smithi158.gmoobf (mgr.14156) 109 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 -- 172.21.15.192:0/214991120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 msgr2=0x7f2ef80f9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 --2- 172.21.15.192:0/214991120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f9210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f2eec009fb0 tx=0x7f2eec009b30).stop 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 -- 172.21.15.192:0/214991120 shutdown_connections 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 --2- 172.21.15.192:0/214991120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f9210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 -- 172.21.15.192:0/214991120 >> 172.21.15.192:0/214991120 conn(0x7f2ef80f0c10 msgr2=0x7f2ef80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 -- 172.21.15.192:0/214991120 shutdown_connections 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.011+0000 7f2efcba4700 1 -- 172.21.15.192:0/214991120 wait complete. 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2efcba4700 1 Processor -- start 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2efcba4700 1 -- start start 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2efcba4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2efcba4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80f5110 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2ef659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2ef659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:33480/0 (socket says 172.21.15.192:33480) 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.012+0000 7f2ef659c700 1 -- 172.21.15.192:0/349161998 learned_addr learned my addr 172.21.15.192:0/349161998 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2ef659c700 1 -- 172.21.15.192:0/349161998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec005040 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2ef659c700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f2eec009f80 tx=0x7f2eec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eec009320 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef80f52b0 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2eec013680 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eec01eda0 con 0x7f2ef80602e0 2024-02-20T13:44:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.013+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef80f5740 con 0x7f2ef80602e0 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2eec01e420 con 0x7f2ef80602e0 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2ee6ffd700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 0x7f2ee0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef80591e0 con 0x7f2ef80602e0 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2eec042360 con 0x7f2ef80602e0 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2ef5d9b700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 0x7f2ee0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:32.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.014+0000 7f2ef5d9b700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 0x7f2ee0033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2ee80096a0 tx=0x7f2ee8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:32.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.019+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2eec01d380 con 0x7f2ef80602e0 2024-02-20T13:44:32.214 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.225+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ef8045bb0 con 0x7f2ef80602e0 2024-02-20T13:44:32.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.226+0000 7f2ee6ffd700 1 -- 172.21.15.192:0/349161998 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2eec02c2d0 con 0x7f2ef80602e0 2024-02-20T13:44:32.215 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:32.216 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 msgr2=0x7f2ee0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 0x7f2ee0033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2ee80096a0 tx=0x7f2ee8006b40).stop 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 msgr2=0x7f2ef80f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f2eec009f80 tx=0x7f2eec00b040).stop 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 shutdown_connections 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ee0031090 0x7f2ee0033550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 --2- 172.21.15.192:0/349161998 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ef80602e0 0x7f2ef80f6c90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 >> 172.21.15.192:0/349161998 conn(0x7f2ef80f0c10 msgr2=0x7f2ef80f16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 shutdown_connections 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:32.228+0000 7f2efcba4700 1 -- 172.21.15.192:0/349161998 wait complete. 2024-02-20T13:44:32.217 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:33.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:31.356976+0000 mgr.smithi158.gmoobf (mgr.14156) 110 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:33.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:32.227583+0000 mon.smithi158 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.192:0/349161998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:33.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:33.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:35.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:33.357234+0000 mgr.smithi158.gmoobf (mgr.14156) 111 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:35.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- 172.21.15.192:0/2638713649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 msgr2=0x7f22f00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:35.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 --2- 172.21.15.192:0/2638713649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f00f56f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f22e4009fb0 tx=0x7f22e4009b30).stop 2024-02-20T13:44:35.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- 172.21.15.192:0/2638713649 shutdown_connections 2024-02-20T13:44:35.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 --2- 172.21.15.192:0/2638713649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f00f56f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- 172.21.15.192:0/2638713649 >> 172.21.15.192:0/2638713649 conn(0x7f22f00f0a30 msgr2=0x7f22f00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- 172.21.15.192:0/2638713649 shutdown_connections 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- 172.21.15.192:0/2638713649 wait complete. 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 Processor -- start 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.793+0000 7f22f58fc700 1 -- start start 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f58fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f58fc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0060360 con 0x7f22f00f52d0 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f48fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f48fa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44184/0 (socket says 172.21.15.192:44184) 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f48fa700 1 -- 172.21.15.192:0/1286336234 learned_addr learned my addr 172.21.15.192:0/1286336234 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.794+0000 7f22f48fa700 1 -- 172.21.15.192:0/1286336234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4005040 con 0x7f22f00f52d0 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22f48fa700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f22e40089f0 tx=0x7f22e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22e4009320 con 0x7f22f00f52d0 2024-02-20T13:44:35.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22e4013ec0 con 0x7f22f00f52d0 2024-02-20T13:44:35.785 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f0060560 con 0x7f22f00f52d0 2024-02-20T13:44:35.785 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22e4014d30 con 0x7f22f00f52d0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.795+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f0109a00 con 0x7f22f00f52d0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f22e4014420 con 0x7f22f00f52d0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22ed7fa700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 0x7f22e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0045bb0 con 0x7f22f00f52d0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f22e4042410 con 0x7f22f00f52d0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22effff700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 0x7f22e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:35.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.796+0000 7f22effff700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 0x7f22e0033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f22dc0096a0 tx=0x7f22dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:35.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:35.801+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f22e4014750 con 0x7f22f00f52d0 2024-02-20T13:44:35.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.005+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22f010a220 con 0x7f22f00f52d0 2024-02-20T13:44:35.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.005+0000 7f22ed7fa700 1 -- 172.21.15.192:0/1286336234 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22e401d7f0 con 0x7f22f00f52d0 2024-02-20T13:44:35.995 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:35.995 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 msgr2=0x7f22e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 0x7f22e0033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f22dc0096a0 tx=0x7f22dc006b40).stop 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 msgr2=0x7f22f01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f22e40089f0 tx=0x7f22e400b040).stop 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 shutdown_connections 2024-02-20T13:44:35.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22e0031090 0x7f22e0033550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 --2- 172.21.15.192:0/1286336234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22f00f52d0 0x7f22f01092d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 >> 172.21.15.192:0/1286336234 conn(0x7f22f00f0a30 msgr2=0x7f22f00f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 shutdown_connections 2024-02-20T13:44:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:36.007+0000 7f22f58fc700 1 -- 172.21.15.192:0/1286336234 wait complete. 2024-02-20T13:44:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:36.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:36.006876+0000 mon.smithi158 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.192:0/1286336234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:37.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:35.357556+0000 mgr.smithi158.gmoobf (mgr.14156) 112 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:37.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:37.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 -- 172.21.15.192:0/3654980230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 msgr2=0x7f00b40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/3654980230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b40f69b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f00a4004760 tx=0x7f00a4009ad0).stop 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 -- 172.21.15.192:0/3654980230 shutdown_connections 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/3654980230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b40f69b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 -- 172.21.15.192:0/3654980230 >> 172.21.15.192:0/3654980230 conn(0x7f00b40f1cd0 msgr2=0x7f00b40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 -- 172.21.15.192:0/3654980230 shutdown_connections 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.409+0000 7f00bb3a2700 1 -- 172.21.15.192:0/3654980230 wait complete. 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00bb3a2700 1 Processor -- start 2024-02-20T13:44:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00bb3a2700 1 -- start start 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00bb3a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00bb3a2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4069f00 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00b913e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00b913e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44200/0 (socket says 172.21.15.192:44200) 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00b913e700 1 -- 172.21.15.192:0/560029536 learned_addr learned my addr 172.21.15.192:0/560029536 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00b913e700 1 -- 172.21.15.192:0/560029536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a4009d70 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00b913e700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f00a4013f10 tx=0x7f00a4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.410+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a400e040 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.411+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b406a0a0 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.411+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f00a4007df0 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.411+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a4014d60 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.411+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b406a530 con 0x7f00b40f6590 2024-02-20T13:44:39.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f00a401f460 con 0x7f00b40f6590 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00a9ffb700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 0x7f00a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f00a4042460 con 0x7f00b40f6590 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b40591e0 con 0x7f00b40f6590 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00b893d700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 0x7f00a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.412+0000 7f00b893d700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 0x7f00a0033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f00b00096a0 tx=0x7f00b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.417+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f00a40283f0 con 0x7f00b40f6590 2024-02-20T13:44:39.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:37.357890+0000 mgr.smithi158.gmoobf (mgr.14156) 113 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:39.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.623+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00b4045bb0 con 0x7f00b40f6590 2024-02-20T13:44:39.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.624+0000 7f00a9ffb700 1 -- 172.21.15.192:0/560029536 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f00a4027e20 con 0x7f00b40f6590 2024-02-20T13:44:39.613 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:39.613 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:39.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 msgr2=0x7f00a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 0x7f00a0033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f00b00096a0 tx=0x7f00b0006b40).stop 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 msgr2=0x7f00b406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f00a4013f10 tx=0x7f00a4009ad0).stop 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 shutdown_connections 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f00a0031090 0x7f00a0033550 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 --2- 172.21.15.192:0/560029536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00b40f6590 0x7f00b406ba80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 >> 172.21.15.192:0/560029536 conn(0x7f00b40f1cd0 msgr2=0x7f00b40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 shutdown_connections 2024-02-20T13:44:39.615 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:39.626+0000 7f00bb3a2700 1 -- 172.21.15.192:0/560029536 wait complete. 2024-02-20T13:44:39.616 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:40.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:39.625641+0000 mon.smithi158 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.192:0/560029536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:41.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:41.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:41.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:39.358190+0000 mgr.smithi158.gmoobf (mgr.14156) 114 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:42.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:41.358448+0000 mgr.smithi158.gmoobf (mgr.14156) 115 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:43.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/1570861823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 msgr2=0x7fc1ac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:43.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/1570861823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac0f69b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc1a0004760 tx=0x7fc1a0009ad0).stop 2024-02-20T13:44:43.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/1570861823 shutdown_connections 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/1570861823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac0f69b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/1570861823 >> 172.21.15.192:0/1570861823 conn(0x7fc1ac0f1cd0 msgr2=0x7fc1ac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/1570861823 shutdown_connections 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.113+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/1570861823 wait complete. 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.114+0000 7fc1b2dcc700 1 Processor -- start 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.114+0000 7fc1b2dcc700 1 -- start start 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.114+0000 7fc1b2dcc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.114+0000 7fc1b2dcc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac106cb0 con 0x7fc1ac0f6590 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b0b68700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b0b68700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51290/0 (socket says 172.21.15.192:51290) 2024-02-20T13:44:43.105 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b0b68700 1 -- 172.21.15.192:0/769143444 learned_addr learned my addr 172.21.15.192:0/769143444 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:43.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b0b68700 1 -- 172.21.15.192:0/769143444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a0009d70 con 0x7fc1ac0f6590 2024-02-20T13:44:43.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b0b68700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fc1a0004170 tx=0x7fc1a0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:43.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a000e040 con 0x7fc1ac0f6590 2024-02-20T13:44:43.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac106eb0 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc1a0008ef0 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a0014c90 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.115+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac107340 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.116+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc1a0014420 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.116+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac045bb0 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.116+0000 7fc1a97fa700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 0x7fc19c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.116+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1a0042220 con 0x7fc1ac0f6590 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.117+0000 7fc1abfff700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 0x7fc19c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:43.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.117+0000 7fc1abfff700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 0x7fc19c033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc1980096a0 tx=0x7fc198006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:43.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.121+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc1a0026300 con 0x7fc1ac0f6590 2024-02-20T13:44:43.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.331+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1ac108190 con 0x7fc1ac0f6590 2024-02-20T13:44:43.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.331+0000 7fc1a97fa700 1 -- 172.21.15.192:0/769143444 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc1a0012b20 con 0x7fc1ac0f6590 2024-02-20T13:44:43.320 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:43.320 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 msgr2=0x7fc19c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 0x7fc19c033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc1980096a0 tx=0x7fc198006b40).stop 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 msgr2=0x7fc1ac1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fc1a0004170 tx=0x7fc1a0009ad0).stop 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 shutdown_connections 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc19c031090 0x7fc19c033550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 --2- 172.21.15.192:0/769143444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1ac0f6590 0x7fc1ac1067c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 >> 172.21.15.192:0/769143444 conn(0x7fc1ac0f1cd0 msgr2=0x7fc1ac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 shutdown_connections 2024-02-20T13:44:43.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:43.333+0000 7fc1b2dcc700 1 -- 172.21.15.192:0/769143444 wait complete. 2024-02-20T13:44:43.323 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:43.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:43.332742+0000 mon.smithi158 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.192:0/769143444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:44.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:44.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:45.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:43.358878+0000 mgr.smithi158.gmoobf (mgr.14156) 116 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:46.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 -- 172.21.15.192:0/4171109177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 msgr2=0x7fbdac0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/4171109177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac0f9160 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbd9c009fb0 tx=0x7fbd9c009b30).stop 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 -- 172.21.15.192:0/4171109177 shutdown_connections 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/4171109177 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac0f9160 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 -- 172.21.15.192:0/4171109177 >> 172.21.15.192:0/4171109177 conn(0x7fbdac0f0be0 msgr2=0x7fbdac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 -- 172.21.15.192:0/4171109177 shutdown_connections 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 -- 172.21.15.192:0/4171109177 wait complete. 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.707+0000 7fbdb0f57700 1 Processor -- start 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdb0f57700 1 -- start start 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdb0f57700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdb0f57700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac106c40 con 0x7fbdac060360 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdaa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:46.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdaa59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51316/0 (socket says 172.21.15.192:51316) 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdaa59c700 1 -- 172.21.15.192:0/614365733 learned_addr learned my addr 172.21.15.192:0/614365733 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdaa59c700 1 -- 172.21.15.192:0/614365733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c005040 con 0x7fbdac060360 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdaa59c700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fbd9c009f80 tx=0x7fbd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9c009320 con 0x7fbdac060360 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.708+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac106e40 con 0x7fbdac060360 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.709+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd9c013680 con 0x7fbdac060360 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.709+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9c01eda0 con 0x7fbdac060360 2024-02-20T13:44:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.709+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac1072d0 con 0x7fbdac060360 2024-02-20T13:44:46.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.709+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdac045bb0 con 0x7fbdac060360 2024-02-20T13:44:46.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.710+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fbd9c01e420 con 0x7fbdac060360 2024-02-20T13:44:46.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.710+0000 7fbd9affd700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 0x7fbd94033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:46.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.710+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd9c042360 con 0x7fbdac060360 2024-02-20T13:44:46.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.710+0000 7fbda9d9b700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 0x7fbd94033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:46.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.710+0000 7fbda9d9b700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 0x7fbd94033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbda00096a0 tx=0x7fbda0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:46.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.714+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd9c01d380 con 0x7fbdac060360 2024-02-20T13:44:46.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.923+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdac108180 con 0x7fbdac060360 2024-02-20T13:44:46.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.924+0000 7fbd9affd700 1 -- 172.21.15.192:0/614365733 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbd9c004490 con 0x7fbdac060360 2024-02-20T13:44:46.913 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:46.913 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:46.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 msgr2=0x7fbd94033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:46.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 0x7fbd94033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbda00096a0 tx=0x7fbda0006b40).stop 2024-02-20T13:44:46.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 msgr2=0x7fbdac106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:46.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fbd9c009f80 tx=0x7fbd9c00b040).stop 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 shutdown_connections 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd94031090 0x7fbd94033550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 --2- 172.21.15.192:0/614365733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbdac060360 0x7fbdac106750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.925+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 >> 172.21.15.192:0/614365733 conn(0x7fbdac0f0be0 msgr2=0x7fbdac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.926+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 shutdown_connections 2024-02-20T13:44:46.915 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:46.926+0000 7fbdb0f57700 1 -- 172.21.15.192:0/614365733 wait complete. 2024-02-20T13:44:46.916 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:45.359316+0000 mgr.smithi158.gmoobf (mgr.14156) 117 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:46.925237+0000 mon.smithi158 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.192:0/614365733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:48.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:48.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:49.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:47.359758+0000 mgr.smithi158.gmoobf (mgr.14156) 118 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- 172.21.15.192:0/467077471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 msgr2=0x7f82240f6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 --2- 172.21.15.192:0/467077471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f82240f6f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8214009fb0 tx=0x7f8214009b30).stop 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- 172.21.15.192:0/467077471 shutdown_connections 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 --2- 172.21.15.192:0/467077471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f82240f6f30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- 172.21.15.192:0/467077471 >> 172.21.15.192:0/467077471 conn(0x7f82240f0390 msgr2=0x7f82240f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- 172.21.15.192:0/467077471 shutdown_connections 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- 172.21.15.192:0/467077471 wait complete. 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 Processor -- start 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.295+0000 7f8228b10700 1 -- start start 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8228b10700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:50.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8228b10700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8224107270 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f822259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f822259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51330/0 (socket says 172.21.15.192:51330) 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f822259c700 1 -- 172.21.15.192:0/3121980899 learned_addr learned my addr 172.21.15.192:0/3121980899 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f822259c700 1 -- 172.21.15.192:0/3121980899 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214005040 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f822259c700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8214009f80 tx=0x7f821400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8214009320 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82240f53a0 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8214013680 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821401eda0 con 0x7f82240f6b10 2024-02-20T13:44:50.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.296+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822410bd70 con 0x7f82240f6b10 2024-02-20T13:44:50.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.297+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f821401e420 con 0x7f82240f6b10 2024-02-20T13:44:50.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.297+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8224045bb0 con 0x7f82240f6b10 2024-02-20T13:44:50.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.297+0000 7f8212ffd700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 0x7f820c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:50.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.297+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f82140423f0 con 0x7f82240f6b10 2024-02-20T13:44:50.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.298+0000 7f8221d9b700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 0x7f820c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:50.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.298+0000 7f8221d9b700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 0x7f820c0335a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f82180096a0 tx=0x7f8218006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:50.291 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.302+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f821401d790 con 0x7f82240f6b10 2024-02-20T13:44:50.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.511+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f822410c530 con 0x7f82240f6b10 2024-02-20T13:44:50.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.512+0000 7f8212ffd700 1 -- 172.21.15.192:0/3121980899 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8214004490 con 0x7f82240f6b10 2024-02-20T13:44:50.500 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:50.501 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 msgr2=0x7f820c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 0x7f820c0335a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f82180096a0 tx=0x7f8218006b40).stop 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 msgr2=0x7f822410b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8214009f80 tx=0x7f821400b040).stop 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 shutdown_connections 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f820c0310e0 0x7f820c0335a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 --2- 172.21.15.192:0/3121980899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82240f6b10 0x7f822410b880 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:50.502 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.513+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 >> 172.21.15.192:0/3121980899 conn(0x7f82240f0390 msgr2=0x7f82240f0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:50.503 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.514+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 shutdown_connections 2024-02-20T13:44:50.503 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:50.514+0000 7f8228b10700 1 -- 172.21.15.192:0/3121980899 wait complete. 2024-02-20T13:44:50.503 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:51.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:49.360128+0000 mgr.smithi158.gmoobf (mgr.14156) 119 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:51.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:50.513404+0000 mon.smithi158 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.192:0/3121980899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:52.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:52.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:53.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:51.360389+0000 mgr.smithi158.gmoobf (mgr.14156) 120 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:53.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 -- 172.21.15.192:0/3535265941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 msgr2=0x7ff3340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:53.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 --2- 172.21.15.192:0/3535265941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3340f69b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ff324004760 tx=0x7ff324009ad0).stop 2024-02-20T13:44:53.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 -- 172.21.15.192:0/3535265941 shutdown_connections 2024-02-20T13:44:53.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 --2- 172.21.15.192:0/3535265941 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3340f69b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:53.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 -- 172.21.15.192:0/3535265941 >> 172.21.15.192:0/3535265941 conn(0x7ff3340f1cd0 msgr2=0x7ff3340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:53.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 -- 172.21.15.192:0/3535265941 shutdown_connections 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.851+0000 7ff33aa11700 1 -- 172.21.15.192:0/3535265941 wait complete. 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff33aa11700 1 Processor -- start 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff33aa11700 1 -- start start 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff33aa11700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff33aa11700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334106cb0 con 0x7ff3340f6590 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff333fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff333fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56984/0 (socket says 172.21.15.192:56984) 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff333fff700 1 -- 172.21.15.192:0/798887623 learned_addr learned my addr 172.21.15.192:0/798887623 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff333fff700 1 -- 172.21.15.192:0/798887623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324009d70 con 0x7ff3340f6590 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.852+0000 7ff333fff700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7ff324004170 tx=0x7ff324009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff32400e040 con 0x7ff3340f6590 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334106eb0 con 0x7ff3340f6590 2024-02-20T13:44:53.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff324008ef0 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff324014c90 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334107340 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.853+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff3240283f0 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.854+0000 7ff330ff9700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 0x7ff320033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.854+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff3240422b0 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.854+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3340591e0 con 0x7ff3340f6590 2024-02-20T13:44:53.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.854+0000 7ff3337fe700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 0x7ff320033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:53.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.854+0000 7ff3337fe700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 0x7ff320033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff31c0096a0 tx=0x7ff31c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:53.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:53.858+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff3240268f0 con 0x7ff3340f6590 2024-02-20T13:44:54.053 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.064+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff334045bb0 con 0x7ff3340f6590 2024-02-20T13:44:54.054 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.065+0000 7ff330ff9700 1 -- 172.21.15.192:0/798887623 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff324026300 con 0x7ff3340f6590 2024-02-20T13:44:54.054 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:54.054 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 msgr2=0x7ff320033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 0x7ff320033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff31c0096a0 tx=0x7ff31c006b40).stop 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 msgr2=0x7ff3341067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7ff324004170 tx=0x7ff324009ad0).stop 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 shutdown_connections 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff320031090 0x7ff320033550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 --2- 172.21.15.192:0/798887623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3340f6590 0x7ff3341067c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 >> 172.21.15.192:0/798887623 conn(0x7ff3340f1cd0 msgr2=0x7ff3340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 shutdown_connections 2024-02-20T13:44:54.056 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:54.067+0000 7ff33aa11700 1 -- 172.21.15.192:0/798887623 wait complete. 2024-02-20T13:44:54.057 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:54.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:54 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:54.066598+0000 mon.smithi158 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.192:0/798887623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:55.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:53.360676+0000 mgr.smithi158.gmoobf (mgr.14156) 121 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:55.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:55.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:55.461861+0000 mon.smithi158 (mon.0) 170 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:44:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:55.462307+0000 mon.smithi158 (mon.0) 171 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:44:57.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- 172.21.15.192:0/1537878333 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 msgr2=0x7fc2bc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:57.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/1537878333 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc0f69b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fc2ac009fb0 tx=0x7fc2ac009b30).stop 2024-02-20T13:44:57.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- 172.21.15.192:0/1537878333 shutdown_connections 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/1537878333 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc0f69b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- 172.21.15.192:0/1537878333 >> 172.21.15.192:0/1537878333 conn(0x7fc2bc0f1cf0 msgr2=0x7fc2bc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- 172.21.15.192:0/1537878333 shutdown_connections 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- 172.21.15.192:0/1537878333 wait complete. 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 Processor -- start 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- start start 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.439+0000 7fc2c45ef700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc106cf0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c238b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c238b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:57002/0 (socket says 172.21.15.192:57002) 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c238b700 1 -- 172.21.15.192:0/2225672742 learned_addr learned my addr 172.21.15.192:0/2225672742 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c238b700 1 -- 172.21.15.192:0/2225672742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac005040 con 0x7fc2bc0f6590 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c238b700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc2ac0089f0 tx=0x7fc2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:57.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2ac009320 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc106ef0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc2ac013ec0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2ac014d30 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc107380 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc2ac014420 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc045bb0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 0x7fc2a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.441+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc2ac042450 con 0x7fc2bc0f6590 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.443+0000 7fc2c1b8a700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 0x7fc2a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.443+0000 7fc2c1b8a700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 0x7fc2a8033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc2b80096a0 tx=0x7fc2b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:44:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.447+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc2ac0263a0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:55.361004+0000 mgr.smithi158.gmoobf (mgr.14156) 122 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:44:57.643 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.656+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2bc0fc080 con 0x7fc2bc0f6590 2024-02-20T13:44:57.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.656+0000 7fc2b2ffd700 1 -- 172.21.15.192:0/2225672742 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2ac01d7f0 con 0x7fc2bc0f6590 2024-02-20T13:44:57.644 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:44:57.644 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:44:57.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 msgr2=0x7fc2a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:57.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 0x7fc2a8033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc2b80096a0 tx=0x7fc2b8006b40).stop 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 msgr2=0x7fc2bc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc2ac0089f0 tx=0x7fc2ac00b040).stop 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 shutdown_connections 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc2a8031090 0x7fc2a8033550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 --2- 172.21.15.192:0/2225672742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2bc0f6590 0x7fc2bc106800 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 >> 172.21.15.192:0/2225672742 conn(0x7fc2bc0f1cf0 msgr2=0x7fc2bc0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 shutdown_connections 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:44:57.658+0000 7fc2c45ef700 1 -- 172.21.15.192:0/2225672742 wait complete. 2024-02-20T13:44:57.646 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:44:58.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:44:57.656848+0000 mon.smithi158 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.192:0/2225672742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:44:59.319 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:44:59.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:44:59.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:44:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:57.361366+0000 mgr.smithi158.gmoobf (mgr.14156) 123 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:01.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- 172.21.15.192:0/3826200957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 msgr2=0x7f13380f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 --2- 172.21.15.192:0/3826200957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13380f58d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f1328009fb0 tx=0x7f1328009b30).stop 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- 172.21.15.192:0/3826200957 shutdown_connections 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 --2- 172.21.15.192:0/3826200957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13380f58d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- 172.21.15.192:0/3826200957 >> 172.21.15.192:0/3826200957 conn(0x7f13380f0c10 msgr2=0x7f13380f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- 172.21.15.192:0/3826200957 shutdown_connections 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- 172.21.15.192:0/3826200957 wait complete. 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 Processor -- start 2024-02-20T13:45:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.109+0000 7f133c8fe700 1 -- start start 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133c8fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133c8fe700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338105b90 con 0x7f13380f54b0 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:57014/0 (socket says 172.21.15.192:57014) 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133659c700 1 -- 172.21.15.192:0/1877992035 learned_addr learned my addr 172.21.15.192:0/1877992035 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133659c700 1 -- 172.21.15.192:0/1877992035 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328005040 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133659c700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f13280089f0 tx=0x7f132800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328009320 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338105d90 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1328013ec0 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328014d30 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.110+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1338106220 con 0x7f13380f54b0 2024-02-20T13:45:01.102 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.111+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1328014420 con 0x7f13380f54b0 2024-02-20T13:45:01.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.111+0000 7f1326ffd700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:01.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.111+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1328042480 con 0x7f13380f54b0 2024-02-20T13:45:01.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.111+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338045bb0 con 0x7f13380f54b0 2024-02-20T13:45:01.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.111+0000 7f1335d9b700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:01.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.112+0000 7f1335d9b700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 0x7f1320033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:01.104 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.116+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1328014750 con 0x7f13380f54b0 2024-02-20T13:45:01.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.327+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1338106ea0 con 0x7f13380f54b0 2024-02-20T13:45:01.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.328+0000 7f1326ffd700 1 -- 172.21.15.192:0/1877992035 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f132802b020 con 0x7f13380f54b0 2024-02-20T13:45:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:01.316 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 msgr2=0x7f1320033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 0x7f1320033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).stop 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 msgr2=0x7f13381056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f13280089f0 tx=0x7f132800b040).stop 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 shutdown_connections 2024-02-20T13:45:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:01.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 --2- 172.21.15.192:0/1877992035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380f54b0 0x7f13381056a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:01.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 >> 172.21.15.192:0/1877992035 conn(0x7f13380f0c10 msgr2=0x7f13380f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:01.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 shutdown_connections 2024-02-20T13:45:01.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:01.329+0000 7f133c8fe700 1 -- 172.21.15.192:0/1877992035 wait complete. 2024-02-20T13:45:01.319 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:01.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:44:59.361699+0000 mgr.smithi158.gmoobf (mgr.14156) 124 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:02.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:01.328154+0000 mon.smithi158 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.192:0/1877992035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:02.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:02.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:03.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:01.362031+0000 mgr.smithi158.gmoobf (mgr.14156) 125 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:04.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 -- 172.21.15.192:0/4023196352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 msgr2=0x7fda680f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/4023196352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda680f56f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fda58009fb0 tx=0x7fda58009b30).stop 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 -- 172.21.15.192:0/4023196352 shutdown_connections 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/4023196352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda680f56f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 -- 172.21.15.192:0/4023196352 >> 172.21.15.192:0/4023196352 conn(0x7fda680f0a30 msgr2=0x7fda680f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 -- 172.21.15.192:0/4023196352 shutdown_connections 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.823+0000 7fda6cb7c700 1 -- 172.21.15.192:0/4023196352 wait complete. 2024-02-20T13:45:04.815 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.824+0000 7fda6cb7c700 1 Processor -- start 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.824+0000 7fda6cb7c700 1 -- start start 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.824+0000 7fda6cb7c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.824+0000 7fda6cb7c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda68107120 con 0x7fda680f52d0 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda677fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39744/0 (socket says 172.21.15.192:39744) 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda677fe700 1 -- 172.21.15.192:0/2686049132 learned_addr learned my addr 172.21.15.192:0/2686049132 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:04.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda677fe700 1 -- 172.21.15.192:0/2686049132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda58005040 con 0x7fda680f52d0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda677fe700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fda580089f0 tx=0x7fda5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda58009320 con 0x7fda680f52d0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda68060300 con 0x7fda680f52d0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fda58013ec0 con 0x7fda680f52d0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda58014d30 con 0x7fda680f52d0 2024-02-20T13:45:04.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.825+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6810bc20 con 0x7fda680f52d0 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.826+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fda580095f0 con 0x7fda680f52d0 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.826+0000 7fda4ffff700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 0x7fda50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.826+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda68045bb0 con 0x7fda680f52d0 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.826+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fda58042450 con 0x7fda680f52d0 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.826+0000 7fda66ffd700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 0x7fda50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.827+0000 7fda66ffd700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 0x7fda50033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:04.819 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:04.831+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fda5801d7f0 con 0x7fda680f52d0 2024-02-20T13:45:05.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.037+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda680faeb0 con 0x7fda680f52d0 2024-02-20T13:45:05.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.038+0000 7fda4ffff700 1 -- 172.21.15.192:0/2686049132 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda5801d7f0 con 0x7fda680f52d0 2024-02-20T13:45:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:05.025 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 msgr2=0x7fda50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 0x7fda50033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c006b40).stop 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 msgr2=0x7fda6810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fda580089f0 tx=0x7fda5800b040).stop 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 shutdown_connections 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda50031090 0x7fda50033550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 --2- 172.21.15.192:0/2686049132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda680f52d0 0x7fda6810b730 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 >> 172.21.15.192:0/2686049132 conn(0x7fda680f0a30 msgr2=0x7fda680f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 shutdown_connections 2024-02-20T13:45:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:05.039+0000 7fda6cb7c700 1 -- 172.21.15.192:0/2686049132 wait complete. 2024-02-20T13:45:05.028 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:05.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:03.362430+0000 mgr.smithi158.gmoobf (mgr.14156) 126 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:05.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:05.038056+0000 mon.smithi158 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.192:0/2686049132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:06.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:06.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:07.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:05.362803+0000 mgr.smithi158.gmoobf (mgr.14156) 127 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:08.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 -- 172.21.15.192:0/2508546360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 msgr2=0x7f4e740f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/2508546360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e740f4d20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4e68009fb0 tx=0x7f4e68009b30).stop 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 -- 172.21.15.192:0/2508546360 shutdown_connections 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/2508546360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e740f4d20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 -- 172.21.15.192:0/2508546360 >> 172.21.15.192:0/2508546360 conn(0x7f4e740f0360 msgr2=0x7f4e740f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 -- 172.21.15.192:0/2508546360 shutdown_connections 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.387+0000 7f4e7ad01700 1 -- 172.21.15.192:0/2508546360 wait complete. 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e7ad01700 1 Processor -- start 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e7ad01700 1 -- start start 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e7ad01700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e7ad01700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74106c90 con 0x7f4e740f4900 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e78a9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e78a9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39774/0 (socket says 172.21.15.192:39774) 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.388+0000 7f4e78a9d700 1 -- 172.21.15.192:0/154658435 learned_addr learned my addr 172.21.15.192:0/154658435 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:08.377 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e78a9d700 1 -- 172.21.15.192:0/154658435 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68005040 con 0x7f4e740f4900 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e78a9d700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4e68009f80 tx=0x7f4e6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e68009320 con 0x7f4e740f4900 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4e68013680 con 0x7f4e740f4900 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74106e90 con 0x7f4e740f4900 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e6801eda0 con 0x7f4e740f4900 2024-02-20T13:45:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.389+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74107320 con 0x7f4e740f4900 2024-02-20T13:45:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.390+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f4e6801e420 con 0x7f4e740f4900 2024-02-20T13:45:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.390+0000 7f4e717fa700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 0x7f4e64033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.390+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74045bb0 con 0x7f4e740f4900 2024-02-20T13:45:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.390+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4e68042300 con 0x7f4e740f4900 2024-02-20T13:45:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.390+0000 7f4e73fff700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 0x7f4e64033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:08.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.391+0000 7f4e73fff700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 0x7f4e64033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4e600096a0 tx=0x7f4e60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:08.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.395+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4e6801ccf0 con 0x7f4e740f4900 2024-02-20T13:45:08.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.603+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e741081d0 con 0x7f4e740f4900 2024-02-20T13:45:08.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.604+0000 7f4e717fa700 1 -- 172.21.15.192:0/154658435 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4e6801cb10 con 0x7f4e740f4900 2024-02-20T13:45:08.592 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:08.592 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:08.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 msgr2=0x7f4e64033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:08.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 0x7f4e64033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4e600096a0 tx=0x7f4e60006b40).stop 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 msgr2=0x7f4e741067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4e68009f80 tx=0x7f4e6800b040).stop 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 shutdown_connections 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4e64031090 0x7f4e64033550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 --2- 172.21.15.192:0/154658435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e740f4900 0x7f4e741067a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:08.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 >> 172.21.15.192:0/154658435 conn(0x7f4e740f0360 msgr2=0x7f4e740f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:08.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 shutdown_connections 2024-02-20T13:45:08.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:08.606+0000 7f4e7ad01700 1 -- 172.21.15.192:0/154658435 wait complete. 2024-02-20T13:45:08.595 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:09.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:07.363142+0000 mgr.smithi158.gmoobf (mgr.14156) 128 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:09.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:08.604297+0000 mon.smithi158 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.192:0/154658435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:10.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:10.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:11.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:09.363495+0000 mgr.smithi158.gmoobf (mgr.14156) 129 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:12.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 -- 172.21.15.192:0/966546942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 msgr2=0x7f8e100f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 --2- 172.21.15.192:0/966546942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e100f58a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8e00009fb0 tx=0x7f8e00009b30).stop 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 -- 172.21.15.192:0/966546942 shutdown_connections 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 --2- 172.21.15.192:0/966546942 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e100f58a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 -- 172.21.15.192:0/966546942 >> 172.21.15.192:0/966546942 conn(0x7f8e100f0be0 msgr2=0x7f8e100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 -- 172.21.15.192:0/966546942 shutdown_connections 2024-02-20T13:45:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.082+0000 7f8e185c1700 1 -- 172.21.15.192:0/966546942 wait complete. 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e185c1700 1 Processor -- start 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e185c1700 1 -- start start 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e185c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e185c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10060300 con 0x7f8e100f5480 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e1635d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:12.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e1635d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39794/0 (socket says 172.21.15.192:39794) 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e1635d700 1 -- 172.21.15.192:0/557740437 learned_addr learned my addr 172.21.15.192:0/557740437 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.083+0000 7f8e1635d700 1 -- 172.21.15.192:0/557740437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00005040 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e1635d700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8e000089f0 tx=0x7f8e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00009320 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e10060500 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e00013ec0 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00014d30 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.084+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10109bc0 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8e00014420 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e100591e0 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e06ffd700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e00042390 con 0x7f8e100f5480 2024-02-20T13:45:12.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e15b5c700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:12.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.085+0000 7f8e15b5c700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 0x7f8dfc033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8e0c0096a0 tx=0x7f8e0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:12.077 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.090+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e0001d7f0 con 0x7f8e100f5480 2024-02-20T13:45:12.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.295+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e10045bb0 con 0x7f8e100f5480 2024-02-20T13:45:12.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.296+0000 7f8e06ffd700 1 -- 172.21.15.192:0/557740437 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e0001d7f0 con 0x7f8e100f5480 2024-02-20T13:45:12.285 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:12.285 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:12.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.297+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 msgr2=0x7f8dfc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:12.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.297+0000 7f8e185c1700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 0x7f8dfc033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8e0c0096a0 tx=0x7f8e0c006b40).stop 2024-02-20T13:45:12.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.297+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 msgr2=0x7f8e10109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:12.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8e000089f0 tx=0x7f8e0000b040).stop 2024-02-20T13:45:12.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 shutdown_connections 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 --2- 172.21.15.192:0/557740437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e100f5480 0x7f8e10109510 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 >> 172.21.15.192:0/557740437 conn(0x7f8e100f0be0 msgr2=0x7f8e100f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 shutdown_connections 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:12.298+0000 7f8e185c1700 1 -- 172.21.15.192:0/557740437 wait complete. 2024-02-20T13:45:12.287 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:13.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:11.363857+0000 mgr.smithi158.gmoobf (mgr.14156) 130 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:13.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:12.297057+0000 mon.smithi158 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.192:0/557740437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:13.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:13.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:15.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:13.364180+0000 mgr.smithi158.gmoobf (mgr.14156) 131 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:15.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 -- 172.21.15.192:0/858317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 msgr2=0x7fbb1c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:15.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 --2- 172.21.15.192:0/858317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c0f58a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fbb04009fb0 tx=0x7fbb04009b30).stop 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 -- 172.21.15.192:0/858317930 shutdown_connections 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 --2- 172.21.15.192:0/858317930 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c0f58a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 -- 172.21.15.192:0/858317930 >> 172.21.15.192:0/858317930 conn(0x7fbb1c0f0be0 msgr2=0x7fbb1c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 -- 172.21.15.192:0/858317930 shutdown_connections 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.697+0000 7fbb220db700 1 -- 172.21.15.192:0/858317930 wait complete. 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb220db700 1 Processor -- start 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb220db700 1 -- start start 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb220db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb220db700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c105be0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb1b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb1b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39068/0 (socket says 172.21.15.192:39068) 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb1b7fe700 1 -- 172.21.15.192:0/1523304961 learned_addr learned my addr 172.21.15.192:0/1523304961 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:15.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.698+0000 7fbb1b7fe700 1 -- 172.21.15.192:0/1523304961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb04005040 con 0x7fbb1c0f5480 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb1b7fe700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fbb040089f0 tx=0x7fbb0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb04009320 con 0x7fbb1c0f5480 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbb04013ec0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c105de0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb04014d30 con 0x7fbb1c0f5480 2024-02-20T13:45:15.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.699+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c106270 con 0x7fbb1c0f5480 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fbb040095f0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c045bb0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb13fff700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 0x7fbb08033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbb04042450 con 0x7fbb1c0f5480 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb1affd700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 0x7fbb08033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:15.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.700+0000 7fbb1affd700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 0x7fbb08033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbb0c0096a0 tx=0x7fbb0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:15.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.704+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbb040268c0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.900 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.910+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb1c0faf70 con 0x7fbb1c0f5480 2024-02-20T13:45:15.900 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.911+0000 7fbb13fff700 1 -- 172.21.15.192:0/1523304961 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb0401d7f0 con 0x7fbb1c0f5480 2024-02-20T13:45:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:15.901 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:15.901 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 msgr2=0x7fbb08033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:15.901 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 0x7fbb08033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbb0c0096a0 tx=0x7fbb0c006b40).stop 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 msgr2=0x7fbb1c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fbb040089f0 tx=0x7fbb0400b040).stop 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 shutdown_connections 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb08031090 0x7fbb08033550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 --2- 172.21.15.192:0/1523304961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 >> 172.21.15.192:0/1523304961 conn(0x7fbb1c0f0be0 msgr2=0x7fbb1c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 shutdown_connections 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:15.912+0000 7fbb220db700 1 -- 172.21.15.192:0/1523304961 wait complete. 2024-02-20T13:45:15.902 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:16.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:15.912005+0000 mon.smithi158 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.192:0/1523304961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:17.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:17.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:17.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:15.364457+0000 mgr.smithi158.gmoobf (mgr.14156) 132 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 -- 172.21.15.192:0/701871804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 msgr2=0x7f81100f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 --2- 172.21.15.192:0/701871804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81100f58d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f8104009fb0 tx=0x7f8104009b30).stop 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 -- 172.21.15.192:0/701871804 shutdown_connections 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 --2- 172.21.15.192:0/701871804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81100f58d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 -- 172.21.15.192:0/701871804 >> 172.21.15.192:0/701871804 conn(0x7f81100f0c10 msgr2=0x7f81100f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 -- 172.21.15.192:0/701871804 shutdown_connections 2024-02-20T13:45:19.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.326+0000 7f8115993700 1 -- 172.21.15.192:0/701871804 wait complete. 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.327+0000 7f8115993700 1 Processor -- start 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.327+0000 7f8115993700 1 -- start start 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.327+0000 7f8115993700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.327+0000 7f8115993700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110105be0 con 0x7f81100f54b0 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f810effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f810effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39090/0 (socket says 172.21.15.192:39090) 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f810effd700 1 -- 172.21.15.192:0/2340367561 learned_addr learned my addr 172.21.15.192:0/2340367561 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f810effd700 1 -- 172.21.15.192:0/2340367561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8104005040 con 0x7f81100f54b0 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f810effd700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8104000c00 tx=0x7f810400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81040095f0 con 0x7f81100f54b0 2024-02-20T13:45:19.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8110105de0 con 0x7f81100f54b0 2024-02-20T13:45:19.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8110106270 con 0x7f81100f54b0 2024-02-20T13:45:19.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.328+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8104007e50 con 0x7f81100f54b0 2024-02-20T13:45:19.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.329+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8104014e00 con 0x7f81100f54b0 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.329+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8104025070 con 0x7f81100f54b0 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.329+0000 7f80f77fe700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 0x7f80f80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.329+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8110045bb0 con 0x7f81100f54b0 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.329+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8104041ce0 con 0x7f81100f54b0 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.330+0000 7f810e7fc700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 0x7f80f80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:19.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.330+0000 7f810e7fc700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 0x7f80f80335a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f81000096a0 tx=0x7f8100006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:19.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.334+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f81040079c0 con 0x7f81100f54b0 2024-02-20T13:45:19.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.541+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81100fb090 con 0x7f81100f54b0 2024-02-20T13:45:19.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.542+0000 7f80f77fe700 1 -- 172.21.15.192:0/2340367561 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f810400f300 con 0x7f81100f54b0 2024-02-20T13:45:19.530 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:19.530 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 msgr2=0x7f80f80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 0x7f80f80335a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f81000096a0 tx=0x7f8100006b40).stop 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 msgr2=0x7f81101056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8104000c00 tx=0x7f810400b040).stop 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 shutdown_connections 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f80f80310e0 0x7f80f80335a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 --2- 172.21.15.192:0/2340367561 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81100f54b0 0x7f81101056f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 >> 172.21.15.192:0/2340367561 conn(0x7f81100f0c10 msgr2=0x7f81100f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:19.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 shutdown_connections 2024-02-20T13:45:19.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:19.543+0000 7f8115993700 1 -- 172.21.15.192:0/2340367561 wait complete. 2024-02-20T13:45:19.533 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:19.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:17.364814+0000 mgr.smithi158.gmoobf (mgr.14156) 133 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:20.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:19.543095+0000 mon.smithi158 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.192:0/2340367561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:21.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:21.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:21.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:19.365184+0000 mgr.smithi158.gmoobf (mgr.14156) 134 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:22.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.976+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/3751258787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 msgr2=0x7f3eec0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.976+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/3751258787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec0f58a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f3edc009fb0 tx=0x7f3edc009b30).stop 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/3751258787 shutdown_connections 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/3751258787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec0f58a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/3751258787 >> 172.21.15.192:0/3751258787 conn(0x7f3eec0f0be0 msgr2=0x7f3eec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/3751258787 shutdown_connections 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/3751258787 wait complete. 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 Processor -- start 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- start start 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.977+0000 7f3ef0ec1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec060300 con 0x7f3eec0f5480 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3eea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:22.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3eea59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54008/0 (socket says 172.21.15.192:54008) 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3eea59c700 1 -- 172.21.15.192:0/115974195 learned_addr learned my addr 172.21.15.192:0/115974195 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3eea59c700 1 -- 172.21.15.192:0/115974195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc005040 con 0x7f3eec0f5480 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3eea59c700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3edc000c00 tx=0x7f3edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc0095f0 con 0x7f3eec0f5480 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec060500 con 0x7f3eec0f5480 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3edc007e50 con 0x7f3eec0f5480 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc014e70 con 0x7f3eec0f5480 2024-02-20T13:45:22.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.978+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec109bc0 con 0x7f3eec0f5480 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.979+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3edc025070 con 0x7f3eec0f5480 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.979+0000 7f3edaffd700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 0x7f3ed4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.979+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec0591e0 con 0x7f3eec0f5480 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.979+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3edc041d10 con 0x7f3eec0f5480 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.980+0000 7f3ee9d9b700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 0x7f3ed4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.980+0000 7f3ee9d9b700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 0x7f3ed4033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3ee00096a0 tx=0x7f3ee0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:22.972 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:22.984+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3edc0079c0 con 0x7f3eec0f5480 2024-02-20T13:45:23.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.193+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3eec0faf70 con 0x7f3eec0f5480 2024-02-20T13:45:23.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.194+0000 7f3edaffd700 1 -- 172.21.15.192:0/115974195 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3edc02d090 con 0x7f3eec0f5480 2024-02-20T13:45:23.182 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:23.182 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:23.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 msgr2=0x7f3ed4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:23.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 0x7f3ed4033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3ee00096a0 tx=0x7f3ee0006b40).stop 2024-02-20T13:45:23.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 msgr2=0x7f3eec109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3edc000c00 tx=0x7f3edc00b040).stop 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 shutdown_connections 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3ed4031090 0x7f3ed4033550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 --2- 172.21.15.192:0/115974195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eec0f5480 0x7f3eec109510 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 >> 172.21.15.192:0/115974195 conn(0x7f3eec0f0be0 msgr2=0x7f3eec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 shutdown_connections 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:23.195+0000 7f3ef0ec1700 1 -- 172.21.15.192:0/115974195 wait complete. 2024-02-20T13:45:23.185 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:21.365471+0000 mgr.smithi158.gmoobf (mgr.14156) 135 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:24.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:23.195037+0000 mon.smithi158 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.192:0/115974195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:24.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:24.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:25.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:23.365828+0000 mgr.smithi158.gmoobf (mgr.14156) 136 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:26.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.716+0000 7f4458cfa700 1 -- 172.21.15.192:0/1545493956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 msgr2=0x7f44540f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.716+0000 7f4458cfa700 1 --2- 172.21.15.192:0/1545493956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44540f58a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f443c009fb0 tx=0x7f443c009b30).stop 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.716+0000 7f4458cfa700 1 -- 172.21.15.192:0/1545493956 shutdown_connections 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.716+0000 7f4458cfa700 1 --2- 172.21.15.192:0/1545493956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44540f58a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.716+0000 7f4458cfa700 1 -- 172.21.15.192:0/1545493956 >> 172.21.15.192:0/1545493956 conn(0x7f44540f0be0 msgr2=0x7f44540f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 -- 172.21.15.192:0/1545493956 shutdown_connections 2024-02-20T13:45:26.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 -- 172.21.15.192:0/1545493956 wait complete. 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 Processor -- start 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 -- start start 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f4458cfa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454105be0 con 0x7f44540f5480 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f445259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:26.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f445259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54028/0 (socket says 172.21.15.192:54028) 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.717+0000 7f445259c700 1 -- 172.21.15.192:0/2758698837 learned_addr learned my addr 172.21.15.192:0/2758698837 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f445259c700 1 -- 172.21.15.192:0/2758698837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c005040 con 0x7f44540f5480 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f445259c700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f443c000c00 tx=0x7f443c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f443c0095f0 con 0x7f44540f5480 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4454105de0 con 0x7f44540f5480 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f443c007e50 con 0x7f44540f5480 2024-02-20T13:45:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f443c014e70 con 0x7f44540f5480 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.718+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4454106270 con 0x7f44540f5480 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.719+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f443c025070 con 0x7f44540f5480 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.719+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454045bb0 con 0x7f44540f5480 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.719+0000 7f444affd700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 0x7f4440033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.719+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f443c041dd0 con 0x7f44540f5480 2024-02-20T13:45:26.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.719+0000 7f4451d9b700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 0x7f4440033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:26.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.720+0000 7f4451d9b700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 0x7f4440033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f44440096a0 tx=0x7f4444006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:26.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.724+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f443c00f300 con 0x7f44540f5480 2024-02-20T13:45:26.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.930+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44540faf70 con 0x7f44540f5480 2024-02-20T13:45:26.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.931+0000 7f444affd700 1 -- 172.21.15.192:0/2758698837 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f443c00f300 con 0x7f44540f5480 2024-02-20T13:45:26.919 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:26.920 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:26.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 msgr2=0x7f4440033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:26.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 0x7f4440033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f44440096a0 tx=0x7f4444006b40).stop 2024-02-20T13:45:26.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 msgr2=0x7f44541056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:26.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f443c000c00 tx=0x7f443c00b040).stop 2024-02-20T13:45:26.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 shutdown_connections 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4440031090 0x7f4440033550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 --2- 172.21.15.192:0/2758698837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44540f5480 0x7f44541056f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 >> 172.21.15.192:0/2758698837 conn(0x7f44540f0be0 msgr2=0x7f44540f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 shutdown_connections 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:26.933+0000 7f4458cfa700 1 -- 172.21.15.192:0/2758698837 wait complete. 2024-02-20T13:45:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:25.366179+0000 mgr.smithi158.gmoobf (mgr.14156) 137 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:26.932735+0000 mon.smithi158 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.192:0/2758698837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:28.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:28.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:29.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:27.366564+0000 mgr.smithi158.gmoobf (mgr.14156) 138 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 -- 172.21.15.192:0/1174696357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e500f6590 msgr2=0x7f3e500f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 --2- 172.21.15.192:0/1174696357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e500f6590 0x7f3e500f69b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f3e40004760 tx=0x7f3e40009ad0).stop 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 -- 172.21.15.192:0/1174696357 shutdown_connections 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 --2- 172.21.15.192:0/1174696357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e500f6590 0x7f3e500f69b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 -- 172.21.15.192:0/1174696357 >> 172.21.15.192:0/1174696357 conn(0x7f3e500f1cd0 msgr2=0x7f3e500f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 -- 172.21.15.192:0/1174696357 shutdown_connections 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 -- 172.21.15.192:0/1174696357 wait complete. 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.416+0000 7f3e5786e700 1 Processor -- start 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5786e700 1 -- start start 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5786e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5786e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5006be60 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5560a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5560a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54058/0 (socket says 172.21.15.192:54058) 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5560a700 1 -- 172.21.15.192:0/2197046545 learned_addr learned my addr 172.21.15.192:0/2197046545 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5560a700 1 -- 172.21.15.192:0/2197046545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40009d70 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e5560a700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3e40009820 tx=0x7f3e40009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.417+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4000e040 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.418+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3e400095b0 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.418+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5006a3f0 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.418+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e40014be0 con 0x7f3e5006ba40 2024-02-20T13:45:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.418+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5006a8e0 con 0x7f3e5006ba40 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.418+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3e40014420 con 0x7f3e5006ba40 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.419+0000 7f3e45ffb700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 0x7f3e3c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.419+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3e40042130 con 0x7f3e5006ba40 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.419+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e500591e0 con 0x7f3e5006ba40 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.419+0000 7f3e54e09700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 0x7f3e3c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.419+0000 7f3e54e09700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 0x7f3e3c033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3e4c0096a0 tx=0x7f3e4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:30.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.424+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3e40014e60 con 0x7f3e5006ba40 2024-02-20T13:45:30.621 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.632+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e50045bb0 con 0x7f3e5006ba40 2024-02-20T13:45:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.633+0000 7f3e45ffb700 1 -- 172.21.15.192:0/2197046545 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3e50045bb0 con 0x7f3e5006ba40 2024-02-20T13:45:30.622 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:30.622 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 msgr2=0x7f3e3c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 0x7f3e3c033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3e4c0096a0 tx=0x7f3e4c006b40).stop 2024-02-20T13:45:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 msgr2=0x7f3e50069f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3e40009820 tx=0x7f3e40009ad0).stop 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 shutdown_connections 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3e3c031090 0x7f3e3c033550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 --2- 172.21.15.192:0/2197046545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5006ba40 0x7f3e50069f00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 >> 172.21.15.192:0/2197046545 conn(0x7f3e500f1cd0 msgr2=0x7f3e500f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 shutdown_connections 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:30.635+0000 7f3e5786e700 1 -- 172.21.15.192:0/2197046545 wait complete. 2024-02-20T13:45:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:31.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:29.366930+0000 mgr.smithi158.gmoobf (mgr.14156) 139 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:31.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:30.634419+0000 mon.smithi158 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.192:0/2197046545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:32.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:32.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:33.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:31.367336+0000 mgr.smithi158.gmoobf (mgr.14156) 140 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- 172.21.15.192:0/2829844887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 msgr2=0x7fca480f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 --2- 172.21.15.192:0/2829844887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca480f58d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fca38009fb0 tx=0x7fca38009b30).stop 2024-02-20T13:45:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- 172.21.15.192:0/2829844887 shutdown_connections 2024-02-20T13:45:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 --2- 172.21.15.192:0/2829844887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca480f58d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- 172.21.15.192:0/2829844887 >> 172.21.15.192:0/2829844887 conn(0x7fca480f0c10 msgr2=0x7fca480f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:34.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- 172.21.15.192:0/2829844887 shutdown_connections 2024-02-20T13:45:34.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- 172.21.15.192:0/2829844887 wait complete. 2024-02-20T13:45:34.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 Processor -- start 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.219+0000 7fca4e39a700 1 -- start start 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca4e39a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca4e39a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48106d10 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca47fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca47fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55222/0 (socket says 172.21.15.192:55222) 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca47fff700 1 -- 172.21.15.192:0/578795308 learned_addr learned my addr 172.21.15.192:0/578795308 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca47fff700 1 -- 172.21.15.192:0/578795308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38005040 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.220+0000 7fca47fff700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fca38000c00 tx=0x7fca3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca380095f0 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca38007e50 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48106f10 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca38014e70 con 0x7fca480f54b0 2024-02-20T13:45:34.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca481073a0 con 0x7fca480f54b0 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fca38025070 con 0x7fca480f54b0 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.221+0000 7fca44ff9700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.222+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fca38041d10 con 0x7fca480f54b0 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.222+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48045bb0 con 0x7fca480f54b0 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.222+0000 7fca477fe700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:34.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.222+0000 7fca477fe700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 0x7fca30033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fca3c0096a0 tx=0x7fca3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:34.214 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.226+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca380079c0 con 0x7fca480f54b0 2024-02-20T13:45:34.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.431+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca480fb0f0 con 0x7fca480f54b0 2024-02-20T13:45:34.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.431+0000 7fca44ff9700 1 -- 172.21.15.192:0/578795308 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca3802d090 con 0x7fca480f54b0 2024-02-20T13:45:34.420 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:34.421 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:34.422 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 msgr2=0x7fca30033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:34.422 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 0x7fca30033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fca3c0096a0 tx=0x7fca3c006b40).stop 2024-02-20T13:45:34.422 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 msgr2=0x7fca48106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:34.422 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fca38000c00 tx=0x7fca3800b040).stop 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 shutdown_connections 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 --2- 172.21.15.192:0/578795308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca480f54b0 0x7fca48106820 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 >> 172.21.15.192:0/578795308 conn(0x7fca480f0c10 msgr2=0x7fca480f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 shutdown_connections 2024-02-20T13:45:34.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:34.433+0000 7fca4e39a700 1 -- 172.21.15.192:0/578795308 wait complete. 2024-02-20T13:45:34.424 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:35.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:33.367628+0000 mgr.smithi158.gmoobf (mgr.14156) 141 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:35.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:34.432894+0000 mon.smithi158 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.192:0/578795308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:36.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:36.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:37.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:35.367972+0000 mgr.smithi158.gmoobf (mgr.14156) 142 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 -- 172.21.15.192:0/3821447708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 msgr2=0x7f681c0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 --2- 172.21.15.192:0/3821447708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c0f4c60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6810009fb0 tx=0x7f6810009b30).stop 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 -- 172.21.15.192:0/3821447708 shutdown_connections 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 --2- 172.21.15.192:0/3821447708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c0f4c60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 -- 172.21.15.192:0/3821447708 >> 172.21.15.192:0/3821447708 conn(0x7f681c0f0360 msgr2=0x7f681c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 -- 172.21.15.192:0/3821447708 shutdown_connections 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.913+0000 7f6820a63700 1 -- 172.21.15.192:0/3821447708 wait complete. 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f6820a63700 1 Processor -- start 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f6820a63700 1 -- start start 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f6820a63700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:37.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f6820a63700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c106cb0 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f681a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f681a59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55248/0 (socket says 172.21.15.192:55248) 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f681a59c700 1 -- 172.21.15.192:0/736914837 learned_addr learned my addr 172.21.15.192:0/736914837 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f681a59c700 1 -- 172.21.15.192:0/736914837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810005040 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f681a59c700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f68100089f0 tx=0x7f681000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6810009320 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c106eb0 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.914+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6810013ec0 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.915+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6810014d30 con 0x7f681c0f4840 2024-02-20T13:45:37.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.915+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c107340 con 0x7f681c0f4840 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.915+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f68100263a0 con 0x7f681c0f4840 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.915+0000 7f680affd700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 0x7f6804033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.915+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6810042480 con 0x7f681c0f4840 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.916+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c0591e0 con 0x7f681c0f4840 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.916+0000 7f6819d9b700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 0x7f6804033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:37.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.916+0000 7f6819d9b700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 0x7f6804033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f680c0096a0 tx=0x7f680c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:37.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:37.920+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6810012920 con 0x7f681c0f4840 2024-02-20T13:45:38.114 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.126+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f681c045bb0 con 0x7f681c0f4840 2024-02-20T13:45:38.115 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.127+0000 7f680affd700 1 -- 172.21.15.192:0/736914837 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f681002b020 con 0x7f681c0f4840 2024-02-20T13:45:38.115 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:38.115 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 msgr2=0x7f6804033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 0x7f6804033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f680c0096a0 tx=0x7f680c006b40).stop 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 msgr2=0x7f681c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f68100089f0 tx=0x7f681000b040).stop 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 shutdown_connections 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6804031090 0x7f6804033550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 --2- 172.21.15.192:0/736914837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f681c0f4840 0x7f681c1067c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 >> 172.21.15.192:0/736914837 conn(0x7f681c0f0360 msgr2=0x7f681c0f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:38.118 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.128+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 shutdown_connections 2024-02-20T13:45:38.118 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:38.129+0000 7f6820a63700 1 -- 172.21.15.192:0/736914837 wait complete. 2024-02-20T13:45:38.118 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:38.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:38.128093+0000 mon.smithi158 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.192:0/736914837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:39.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:37.368327+0000 mgr.smithi158.gmoobf (mgr.14156) 143 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:39.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:39.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:41.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:39.368679+0000 mgr.smithi158.gmoobf (mgr.14156) 144 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:41.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 -- 172.21.15.192:0/714413284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 msgr2=0x7f4a600f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 --2- 172.21.15.192:0/714413284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a600f58d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f4a50009fb0 tx=0x7f4a50009b30).stop 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 -- 172.21.15.192:0/714413284 shutdown_connections 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 --2- 172.21.15.192:0/714413284 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a600f58d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 -- 172.21.15.192:0/714413284 >> 172.21.15.192:0/714413284 conn(0x7f4a600f0c10 msgr2=0x7f4a600f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 -- 172.21.15.192:0/714413284 shutdown_connections 2024-02-20T13:45:41.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.549+0000 7f4a6593b700 1 -- 172.21.15.192:0/714413284 wait complete. 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a6593b700 1 Processor -- start 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a6593b700 1 -- start start 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a6593b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a6593b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60060320 con 0x7f4a600f54b0 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a5effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a5effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55262/0 (socket says 172.21.15.192:55262) 2024-02-20T13:45:41.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.550+0000 7f4a5effd700 1 -- 172.21.15.192:0/3896313991 learned_addr learned my addr 172.21.15.192:0/3896313991 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a5effd700 1 -- 172.21.15.192:0/3896313991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a50005040 con 0x7f4a600f54b0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a5effd700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f4a50000c00 tx=0x7f4a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a500095f0 con 0x7f4a600f54b0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a60060520 con 0x7f4a600f54b0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4a50007e50 con 0x7f4a600f54b0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a50014e70 con 0x7f4a600f54b0 2024-02-20T13:45:41.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.551+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a60109b50 con 0x7f4a600f54b0 2024-02-20T13:45:41.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.552+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f4a50025070 con 0x7f4a600f54b0 2024-02-20T13:45:41.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.552+0000 7f4a477fe700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 0x7f4a48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:41.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.552+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4a50041dc0 con 0x7f4a600f54b0 2024-02-20T13:45:41.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.552+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a600591e0 con 0x7f4a600f54b0 2024-02-20T13:45:41.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.552+0000 7f4a5e7fc700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 0x7f4a48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:41.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.553+0000 7f4a5e7fc700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 0x7f4a48033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4a540096a0 tx=0x7f4a54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:41.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.557+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a50026350 con 0x7f4a600f54b0 2024-02-20T13:45:41.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.761+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a60045bb0 con 0x7f4a600f54b0 2024-02-20T13:45:41.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.762+0000 7f4a477fe700 1 -- 172.21.15.192:0/3896313991 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a5000f300 con 0x7f4a600f54b0 2024-02-20T13:45:41.750 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 msgr2=0x7f4a48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 0x7f4a48033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4a540096a0 tx=0x7f4a54006b40).stop 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 msgr2=0x7f4a601094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f4a50000c00 tx=0x7f4a5000b040).stop 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 shutdown_connections 2024-02-20T13:45:41.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a48031090 0x7f4a48033550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:41.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 --2- 172.21.15.192:0/3896313991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a600f54b0 0x7f4a601094a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:41.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 >> 172.21.15.192:0/3896313991 conn(0x7f4a600f0c10 msgr2=0x7f4a600f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:41.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 shutdown_connections 2024-02-20T13:45:41.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:41.764+0000 7f4a6593b700 1 -- 172.21.15.192:0/3896313991 wait complete. 2024-02-20T13:45:41.753 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:42.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:41.763564+0000 mon.smithi158 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.192:0/3896313991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:43.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:43.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:43.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:41.368938+0000 mgr.smithi158.gmoobf (mgr.14156) 145 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:45.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.163+0000 7f1838e3e700 1 -- 172.21.15.192:0/4019740172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 msgr2=0x7f18340f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.163+0000 7f1838e3e700 1 --2- 172.21.15.192:0/4019740172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f18340f6800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f1824009fb0 tx=0x7f1824009b30).stop 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.163+0000 7f1838e3e700 1 -- 172.21.15.192:0/4019740172 shutdown_connections 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.163+0000 7f1838e3e700 1 --2- 172.21.15.192:0/4019740172 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f18340f6800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.163+0000 7f1838e3e700 1 -- 172.21.15.192:0/4019740172 >> 172.21.15.192:0/4019740172 conn(0x7f18340f1b40 msgr2=0x7f18340f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 -- 172.21.15.192:0/4019740172 shutdown_connections 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 -- 172.21.15.192:0/4019740172 wait complete. 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 Processor -- start 2024-02-20T13:45:45.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 -- start start 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.164+0000 7f1838e3e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1834106b10 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f18337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f18337fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48492/0 (socket says 172.21.15.192:48492) 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f18337fe700 1 -- 172.21.15.192:0/1115531009 learned_addr learned my addr 172.21.15.192:0/1115531009 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f18337fe700 1 -- 172.21.15.192:0/1115531009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824005040 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f18337fe700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f18240089f0 tx=0x7f182400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1824009320 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1834106d10 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1824013ec0 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1824014d30 con 0x7f18340f63e0 2024-02-20T13:45:45.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.165+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18341071a0 con 0x7f18340f63e0 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.166+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f18240095f0 con 0x7f18340f63e0 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.166+0000 7f181bfff700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 0x7f181c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.166+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1834045bb0 con 0x7f18340f63e0 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.166+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1824042450 con 0x7f18340f63e0 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.167+0000 7f1832ffd700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 0x7f181c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.167+0000 7f1832ffd700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 0x7f181c033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f18280096a0 tx=0x7f1828006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:45.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.172+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f182401d7f0 con 0x7f18340f63e0 2024-02-20T13:45:45.367 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.379+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18340fbfc0 con 0x7f18340f63e0 2024-02-20T13:45:45.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.380+0000 7f181bfff700 1 -- 172.21.15.192:0/1115531009 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f182401d7f0 con 0x7f18340f63e0 2024-02-20T13:45:45.368 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:45.368 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:45.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 msgr2=0x7f181c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:45.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 0x7f181c033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f18280096a0 tx=0x7f1828006b40).stop 2024-02-20T13:45:45.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 msgr2=0x7f1834106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f18240089f0 tx=0x7f182400b040).stop 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 shutdown_connections 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f181c031090 0x7f181c033550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 --2- 172.21.15.192:0/1115531009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18340f63e0 0x7f1834106620 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 >> 172.21.15.192:0/1115531009 conn(0x7f18340f1b40 msgr2=0x7f18340f2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 shutdown_connections 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:45.381+0000 7f1838e3e700 1 -- 172.21.15.192:0/1115531009 wait complete. 2024-02-20T13:45:45.370 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:45.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:43.369214+0000 mgr.smithi158.gmoobf (mgr.14156) 146 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:46.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:45.381179+0000 mon.smithi158 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.192:0/1115531009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:47.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:47.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:47.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:45.369492+0000 mgr.smithi158.gmoobf (mgr.14156) 147 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- 172.21.15.192:0/735836566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 msgr2=0x7f35c40f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 --2- 172.21.15.192:0/735836566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c40f6800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f35b4009fb0 tx=0x7f35b4009b30).stop 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- 172.21.15.192:0/735836566 shutdown_connections 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 --2- 172.21.15.192:0/735836566 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c40f6800 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- 172.21.15.192:0/735836566 >> 172.21.15.192:0/735836566 conn(0x7f35c40f1b40 msgr2=0x7f35c40f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- 172.21.15.192:0/735836566 shutdown_connections 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- 172.21.15.192:0/735836566 wait complete. 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 Processor -- start 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.007+0000 7f35c9186700 1 -- start start 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c9186700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:48.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c9186700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4106b10 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48512/0 (socket says 172.21.15.192:48512) 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c3fff700 1 -- 172.21.15.192:0/3677453323 learned_addr learned my addr 172.21.15.192:0/3677453323 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c3fff700 1 -- 172.21.15.192:0/3677453323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4005040 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c3fff700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f35b40089f0 tx=0x7f35b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b4009320 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f35b4013ec0 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c4106d10 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b4014d30 con 0x7f35c40f63e0 2024-02-20T13:45:48.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.008+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c41071a0 con 0x7f35c40f63e0 2024-02-20T13:45:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.009+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f35b40263a0 con 0x7f35c40f63e0 2024-02-20T13:45:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.009+0000 7f35c0ff9700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 0x7f35ac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.009+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f35b40423c0 con 0x7f35c40f63e0 2024-02-20T13:45:48.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.009+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4045bb0 con 0x7f35c40f63e0 2024-02-20T13:45:48.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.010+0000 7f35c37fe700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 0x7f35ac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:48.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.010+0000 7f35c37fe700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 0x7f35ac033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f35b80096a0 tx=0x7f35b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:49.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.014+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f35b40266d0 con 0x7f35c40f63e0 2024-02-20T13:45:49.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.222+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35c4107e20 con 0x7f35c40f63e0 2024-02-20T13:45:49.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.223+0000 7f35c0ff9700 1 -- 172.21.15.192:0/3677453323 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f35b402b020 con 0x7f35c40f63e0 2024-02-20T13:45:49.210 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:49.210 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:49.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 msgr2=0x7f35ac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:49.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 0x7f35ac033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f35b80096a0 tx=0x7f35b8006b40).stop 2024-02-20T13:45:49.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 msgr2=0x7f35c4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f35b40089f0 tx=0x7f35b400b040).stop 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 shutdown_connections 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f35ac031090 0x7f35ac033550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 --2- 172.21.15.192:0/3677453323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35c40f63e0 0x7f35c4106620 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 >> 172.21.15.192:0/3677453323 conn(0x7f35c40f1b40 msgr2=0x7f35c40f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 shutdown_connections 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:49.224+0000 7f35c9186700 1 -- 172.21.15.192:0/3677453323 wait complete. 2024-02-20T13:45:49.212 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:49.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:47.369807+0000 mgr.smithi158.gmoobf (mgr.14156) 148 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:50.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:49.223052+0000 mon.smithi158 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.192:0/3677453323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:50.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:50.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:51.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:49.370053+0000 mgr.smithi158.gmoobf (mgr.14156) 149 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:52.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.724+0000 7fc3d67fc700 1 -- 172.21.15.192:0/1483456222 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c8013860 con 0x7fc3d80f5480 2024-02-20T13:45:52.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 -- 172.21.15.192:0/1483456222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 msgr2=0x7fc3d80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:52.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 --2- 172.21.15.192:0/1483456222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d80f58a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc3c8009fb0 tx=0x7fc3c8009b30).stop 2024-02-20T13:45:52.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 -- 172.21.15.192:0/1483456222 shutdown_connections 2024-02-20T13:45:52.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 --2- 172.21.15.192:0/1483456222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d80f58a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:52.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 -- 172.21.15.192:0/1483456222 >> 172.21.15.192:0/1483456222 conn(0x7fc3d80f0be0 msgr2=0x7fc3d80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:52.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 -- 172.21.15.192:0/1483456222 shutdown_connections 2024-02-20T13:45:52.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 -- 172.21.15.192:0/1483456222 wait complete. 2024-02-20T13:45:52.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.725+0000 7fc3dde81700 1 Processor -- start 2024-02-20T13:45:52.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3dde81700 1 -- start start 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3dde81700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3dde81700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8104ad0 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56806/0 (socket says 172.21.15.192:56806) 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3d77fe700 1 -- 172.21.15.192:0/2689630730 learned_addr learned my addr 172.21.15.192:0/2689630730 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.726+0000 7fc3d77fe700 1 -- 172.21.15.192:0/2689630730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8005040 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3d77fe700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fc3c80089f0 tx=0x7fc3c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c8009320 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d8104cd0 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc3c8013ec0 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c8014d30 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.727+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d8105160 con 0x7fc3d80f5480 2024-02-20T13:45:52.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc3c80263a0 con 0x7fc3d80f5480 2024-02-20T13:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3bffff700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 0x7fc3c0037a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8045bb0 con 0x7fc3d80f5480 2024-02-20T13:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc3c8042410 con 0x7fc3d80f5480 2024-02-20T13:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3d6ffd700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 0x7fc3c0037a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.728+0000 7fc3d6ffd700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 0x7fc3c0037a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc3cc0096a0 tx=0x7fc3cc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:52.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.732+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc3c801cb10 con 0x7fc3d80f5480 2024-02-20T13:45:52.927 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.940+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3d8105f60 con 0x7fc3d80f5480 2024-02-20T13:45:52.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.941+0000 7fc3bffff700 1 -- 172.21.15.192:0/2689630730 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc3c802b020 con 0x7fc3d80f5480 2024-02-20T13:45:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:52.928 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 msgr2=0x7fc3c0037a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 0x7fc3c0037a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc3cc0096a0 tx=0x7fc3cc006b40).stop 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 msgr2=0x7fc3d81045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fc3c80089f0 tx=0x7fc3c800b040).stop 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 shutdown_connections 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc3c0035540 0x7fc3c0037a00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 --2- 172.21.15.192:0/2689630730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3d80f5480 0x7fc3d81045e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 >> 172.21.15.192:0/2689630730 conn(0x7fc3d80f0be0 msgr2=0x7fc3d80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 shutdown_connections 2024-02-20T13:45:52.930 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:52.943+0000 7fc3dde81700 1 -- 172.21.15.192:0/2689630730 wait complete. 2024-02-20T13:45:52.931 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:53.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:51.370402+0000 mgr.smithi158.gmoobf (mgr.14156) 150 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:53.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:52.941581+0000 mon.smithi158 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.192:0/2689630730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:54.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:54.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:55.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:53.370799+0000 mgr.smithi158.gmoobf (mgr.14156) 151 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 -- 172.21.15.192:0/1279452052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 msgr2=0x7f9bd80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/1279452052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd80f56f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f9bc8009fb0 tx=0x7f9bc8009b30).stop 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 -- 172.21.15.192:0/1279452052 shutdown_connections 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/1279452052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd80f56f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 -- 172.21.15.192:0/1279452052 >> 172.21.15.192:0/1279452052 conn(0x7f9bd80f0a30 msgr2=0x7f9bd80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 -- 172.21.15.192:0/1279452052 shutdown_connections 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.472+0000 7f9bdd7db700 1 -- 172.21.15.192:0/1279452052 wait complete. 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bdd7db700 1 Processor -- start 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bdd7db700 1 -- start start 2024-02-20T13:45:56.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bdd7db700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:56.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bdd7db700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8060300 con 0x7f9bd80f52d0 2024-02-20T13:45:56.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bd7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:56.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bd7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56824/0 (socket says 172.21.15.192:56824) 2024-02-20T13:45:56.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bd7fff700 1 -- 172.21.15.192:0/3163985945 learned_addr learned my addr 172.21.15.192:0/3163985945 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:45:56.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bd7fff700 1 -- 172.21.15.192:0/3163985945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8005040 con 0x7f9bd80f52d0 2024-02-20T13:45:56.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.473+0000 7f9bd7fff700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9bc80089f0 tx=0x7f9bc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:56.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc8009320 con 0x7f9bd80f52d0 2024-02-20T13:45:56.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd8060500 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9bc8013ec0 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc8014d30 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81099e0 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.474+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9bc8014420 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.475+0000 7f9bd57fa700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 0x7f9bc4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.475+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9bc8042390 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.475+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd80591e0 con 0x7f9bd80f52d0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.475+0000 7f9bcffff700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 0x7f9bc4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:45:56.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.475+0000 7f9bcffff700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 0x7f9bc4033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9bc00096a0 tx=0x7f9bc0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:45:56.466 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.480+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9bc8014750 con 0x7f9bd80f52d0 2024-02-20T13:45:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:55.463020+0000 mon.smithi158 (mon.0) 188 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:45:56.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:55.463250+0000 mon.smithi158 (mon.0) 189 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:45:56.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.684+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bd8045bb0 con 0x7f9bd80f52d0 2024-02-20T13:45:56.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.684+0000 7f9bd57fa700 1 -- 172.21.15.192:0/3163985945 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9bd8045bb0 con 0x7f9bd80f52d0 2024-02-20T13:45:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:45:56.672 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 msgr2=0x7f9bc4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 0x7f9bc4033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9bc00096a0 tx=0x7f9bc0006b40).stop 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 msgr2=0x7f9bd8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9bc80089f0 tx=0x7f9bc800b040).stop 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 shutdown_connections 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9bc4031090 0x7f9bc4033550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 --2- 172.21.15.192:0/3163985945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd80f52d0 0x7f9bd8109330 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 >> 172.21.15.192:0/3163985945 conn(0x7f9bd80f0a30 msgr2=0x7f9bd80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:45:56.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 shutdown_connections 2024-02-20T13:45:56.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:45:56.686+0000 7f9bdd7db700 1 -- 172.21.15.192:0/3163985945 wait complete. 2024-02-20T13:45:56.674 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:45:57.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:55.371071+0000 mgr.smithi158.gmoobf (mgr.14156) 152 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:45:57.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:45:56.684886+0000 mon.smithi158 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.192:0/3163985945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:45:58.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:45:58.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:45:59.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:45:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:57.371386+0000 mgr.smithi158.gmoobf (mgr.14156) 153 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:00.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 -- 172.21.15.192:0/1985663080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 msgr2=0x7eff500f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 --2- 172.21.15.192:0/1985663080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff500f6eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7eff44009fb0 tx=0x7eff44009b30).stop 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 -- 172.21.15.192:0/1985663080 shutdown_connections 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 --2- 172.21.15.192:0/1985663080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff500f6eb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 -- 172.21.15.192:0/1985663080 >> 172.21.15.192:0/1985663080 conn(0x7eff500f0360 msgr2=0x7eff500f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 -- 172.21.15.192:0/1985663080 shutdown_connections 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.194+0000 7eff54915700 1 -- 172.21.15.192:0/1985663080 wait complete. 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff54915700 1 Processor -- start 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff54915700 1 -- start start 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff54915700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff54915700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff50105bf0 con 0x7eff500f6a90 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff4e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:00.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff4e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56838/0 (socket says 172.21.15.192:56838) 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.195+0000 7eff4e59c700 1 -- 172.21.15.192:0/2244001840 learned_addr learned my addr 172.21.15.192:0/2244001840 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff4e59c700 1 -- 172.21.15.192:0/2244001840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44005040 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff4e59c700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7eff440089f0 tx=0x7eff4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff44009320 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff50105df0 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7eff44013ec0 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff44014d30 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.196+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff50106280 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7eff440095f0 con 0x7eff500f6a90 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff3effd700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 0x7eff38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff50045bb0 con 0x7eff500f6a90 2024-02-20T13:46:00.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7eff44042390 con 0x7eff500f6a90 2024-02-20T13:46:00.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff4dd9b700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 0x7eff38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:00.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.197+0000 7eff4dd9b700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 0x7eff38033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7eff400096a0 tx=0x7eff40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:00.199 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.202+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7eff4400fe40 con 0x7eff500f6a90 2024-02-20T13:46:00.399 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.411+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff50107130 con 0x7eff500f6a90 2024-02-20T13:46:00.400 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.412+0000 7eff3effd700 1 -- 172.21.15.192:0/2244001840 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7eff4400fe40 con 0x7eff500f6a90 2024-02-20T13:46:00.400 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:00.400 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 msgr2=0x7eff38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 0x7eff38033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7eff400096a0 tx=0x7eff40006b40).stop 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 msgr2=0x7eff50105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7eff440089f0 tx=0x7eff4400b040).stop 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 shutdown_connections 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7eff38031090 0x7eff38033550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 --2- 172.21.15.192:0/2244001840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff500f6a90 0x7eff50105700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 >> 172.21.15.192:0/2244001840 conn(0x7eff500f0360 msgr2=0x7eff500f0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 shutdown_connections 2024-02-20T13:46:00.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:00.414+0000 7eff54915700 1 -- 172.21.15.192:0/2244001840 wait complete. 2024-02-20T13:46:00.403 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:01.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:45:59.371691+0000 mgr.smithi158.gmoobf (mgr.14156) 154 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:01.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:00.413468+0000 mon.smithi158 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.192:0/2244001840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:02.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:02.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:03.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:01.372021+0000 mgr.smithi158.gmoobf (mgr.14156) 155 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:03.918 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 -- 172.21.15.192:0/577240188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 msgr2=0x7fca040f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/577240188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca040f58d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc9f8009fb0 tx=0x7fc9f8009b30).stop 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 -- 172.21.15.192:0/577240188 shutdown_connections 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/577240188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca040f58d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 -- 172.21.15.192:0/577240188 >> 172.21.15.192:0/577240188 conn(0x7fca040f0c10 msgr2=0x7fca040f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.929+0000 7fca0c5fd700 1 -- 172.21.15.192:0/577240188 shutdown_connections 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.930+0000 7fca0c5fd700 1 -- 172.21.15.192:0/577240188 wait complete. 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.930+0000 7fca0c5fd700 1 Processor -- start 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.930+0000 7fca0c5fd700 1 -- start start 2024-02-20T13:46:03.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.930+0000 7fca0c5fd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:03.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.930+0000 7fca0c5fd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca04106d20 con 0x7fca040f54b0 2024-02-20T13:46:03.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fca0a399700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:03.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fca0a399700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:59570/0 (socket says 172.21.15.192:59570) 2024-02-20T13:46:03.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fca0a399700 1 -- 172.21.15.192:0/4136059968 learned_addr learned my addr 172.21.15.192:0/4136059968 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fca0a399700 1 -- 172.21.15.192:0/4136059968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f8005040 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fca0a399700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc9f80089f0 tx=0x7fc9f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.931+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9f8009320 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc9f8013ec0 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca04106f20 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9f8014d30 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca041073b0 con 0x7fca040f54b0 2024-02-20T13:46:03.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc9f8014420 con 0x7fca040f54b0 2024-02-20T13:46:03.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fc9f6ffd700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 0x7fc9f00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:03.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.932+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca04045bb0 con 0x7fca040f54b0 2024-02-20T13:46:03.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.933+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc9f8042370 con 0x7fca040f54b0 2024-02-20T13:46:03.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.933+0000 7fca09b98700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 0x7fc9f00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:03.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.933+0000 7fca09b98700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 0x7fc9f00335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fca000096a0 tx=0x7fca00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:03.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:03.937+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc9f801d980 con 0x7fca040f54b0 2024-02-20T13:46:04.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.147+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca041081b0 con 0x7fca040f54b0 2024-02-20T13:46:04.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.147+0000 7fc9f6ffd700 1 -- 172.21.15.192:0/4136059968 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc9f802c090 con 0x7fca040f54b0 2024-02-20T13:46:04.135 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:04.135 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 msgr2=0x7fc9f00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 0x7fc9f00335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fca000096a0 tx=0x7fca00006b40).stop 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 msgr2=0x7fca04106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc9f80089f0 tx=0x7fc9f800b040).stop 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 shutdown_connections 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc9f00310e0 0x7fc9f00335a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 --2- 172.21.15.192:0/4136059968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca040f54b0 0x7fca04106830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 >> 172.21.15.192:0/4136059968 conn(0x7fca040f0c10 msgr2=0x7fca040f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 shutdown_connections 2024-02-20T13:46:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:04.149+0000 7fca0c5fd700 1 -- 172.21.15.192:0/4136059968 wait complete. 2024-02-20T13:46:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:05.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:03.372355+0000 mgr.smithi158.gmoobf (mgr.14156) 156 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:05.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:04.148575+0000 mon.smithi158 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.192:0/4136059968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:05.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:05.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:07.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:05.372718+0000 mgr.smithi158.gmoobf (mgr.14156) 157 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:07.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 -- 172.21.15.192:0/603955403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 msgr2=0x7f0ab80f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:07.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 --2- 172.21.15.192:0/603955403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab80f4c60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0aa0009fb0 tx=0x7f0aa0009b30).stop 2024-02-20T13:46:07.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 -- 172.21.15.192:0/603955403 shutdown_connections 2024-02-20T13:46:07.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 --2- 172.21.15.192:0/603955403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab80f4c60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:07.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 -- 172.21.15.192:0/603955403 >> 172.21.15.192:0/603955403 conn(0x7f0ab80f0360 msgr2=0x7f0ab80f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 -- 172.21.15.192:0/603955403 shutdown_connections 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.637+0000 7f0abe443700 1 -- 172.21.15.192:0/603955403 wait complete. 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0abe443700 1 Processor -- start 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0abe443700 1 -- start start 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0abe443700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0abe443700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8106cb0 con 0x7f0ab80f4840 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0ab7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0ab7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:59588/0 (socket says 172.21.15.192:59588) 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.638+0000 7f0ab7fff700 1 -- 172.21.15.192:0/685461579 learned_addr learned my addr 172.21.15.192:0/685461579 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0ab7fff700 1 -- 172.21.15.192:0/685461579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa0005040 con 0x7f0ab80f4840 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0ab7fff700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0aa0000c00 tx=0x7f0aa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aa00095f0 con 0x7f0ab80f4840 2024-02-20T13:46:07.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab8106eb0 con 0x7f0ab80f4840 2024-02-20T13:46:07.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0aa0007e50 con 0x7f0ab80f4840 2024-02-20T13:46:07.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aa0014e70 con 0x7f0ab80f4840 2024-02-20T13:46:07.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.639+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab8107340 con 0x7f0ab80f4840 2024-02-20T13:46:07.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.640+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f0aa0025070 con 0x7f0ab80f4840 2024-02-20T13:46:07.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.640+0000 7f0ab4ff9700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 0x7f0aa40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:07.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.640+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0aa0041df0 con 0x7f0ab80f4840 2024-02-20T13:46:07.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.640+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab8045bb0 con 0x7f0ab80f4840 2024-02-20T13:46:07.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.640+0000 7f0ab77fe700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 0x7f0aa40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:07.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.641+0000 7f0ab77fe700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 0x7f0aa40335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0aa80096a0 tx=0x7f0aa8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:07.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.645+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0aa0026350 con 0x7f0ab80f4840 2024-02-20T13:46:07.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.853+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ab8107ff0 con 0x7f0ab80f4840 2024-02-20T13:46:07.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.854+0000 7f0ab4ff9700 1 -- 172.21.15.192:0/685461579 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0aa000f300 con 0x7f0ab80f4840 2024-02-20T13:46:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:07.842 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 msgr2=0x7f0aa40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 0x7f0aa40335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0aa80096a0 tx=0x7f0aa8006b40).stop 2024-02-20T13:46:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 msgr2=0x7f0ab81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0aa0000c00 tx=0x7f0aa000b040).stop 2024-02-20T13:46:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 shutdown_connections 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0aa40310e0 0x7f0aa40335a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 --2- 172.21.15.192:0/685461579 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ab80f4840 0x7f0ab81067c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 >> 172.21.15.192:0/685461579 conn(0x7f0ab80f0360 msgr2=0x7f0ab80f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 shutdown_connections 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:07.856+0000 7f0abe443700 1 -- 172.21.15.192:0/685461579 wait complete. 2024-02-20T13:46:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:08.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:08 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:07.855447+0000 mon.smithi158 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.192:0/685461579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:09.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:09.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:09.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:07.373037+0000 mgr.smithi158.gmoobf (mgr.14156) 158 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 -- 172.21.15.192:0/1775318555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 msgr2=0x7f72180f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 --2- 172.21.15.192:0/1775318555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72180f58a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7208009fb0 tx=0x7f7208009b30).stop 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 -- 172.21.15.192:0/1775318555 shutdown_connections 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 --2- 172.21.15.192:0/1775318555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72180f58a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 -- 172.21.15.192:0/1775318555 >> 172.21.15.192:0/1775318555 conn(0x7f72180f0be0 msgr2=0x7f72180f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 -- 172.21.15.192:0/1775318555 shutdown_connections 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.173+0000 7f721fa38700 1 -- 172.21.15.192:0/1775318555 wait complete. 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721fa38700 1 Processor -- start 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721fa38700 1 -- start start 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721fa38700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721fa38700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7218060360 con 0x7f72180f5480 2024-02-20T13:46:11.163 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721d7d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721d7d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:59604/0 (socket says 172.21.15.192:59604) 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721d7d4700 1 -- 172.21.15.192:0/463778918 learned_addr learned my addr 172.21.15.192:0/463778918 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721d7d4700 1 -- 172.21.15.192:0/463778918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208005040 con 0x7f72180f5480 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.174+0000 7f721d7d4700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f72080089f0 tx=0x7f720800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.175+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7208009320 con 0x7f72180f5480 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.175+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7218060560 con 0x7f72180f5480 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.175+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7208013ec0 con 0x7f72180f5480 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.175+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7208014d30 con 0x7f72180f5480 2024-02-20T13:46:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.175+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7218109be0 con 0x7f72180f5480 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f72080263a0 con 0x7f72180f5480 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f720dffb700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 0x7f7204033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f72080424d0 con 0x7f72180f5480 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72180591e0 con 0x7f72180f5480 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f721cfd3700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 0x7f7204033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.176+0000 7f721cfd3700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 0x7f7204033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f72140096a0 tx=0x7f7214006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.181+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f72080266d0 con 0x7f72180f5480 2024-02-20T13:46:11.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.385+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72180faf70 con 0x7f72180f5480 2024-02-20T13:46:11.373 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.385+0000 7f720dffb700 1 -- 172.21.15.192:0/463778918 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f720802b020 con 0x7f72180f5480 2024-02-20T13:46:11.373 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:11.373 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 msgr2=0x7f7204033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 0x7f7204033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f72140096a0 tx=0x7f7214006b40).stop 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 msgr2=0x7f72181094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f72080089f0 tx=0x7f720800b040).stop 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 shutdown_connections 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7204031090 0x7f7204033550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 --2- 172.21.15.192:0/463778918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72180f5480 0x7f72181094b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 >> 172.21.15.192:0/463778918 conn(0x7f72180f0be0 msgr2=0x7f72180f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 shutdown_connections 2024-02-20T13:46:11.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:11.387+0000 7f721fa38700 1 -- 172.21.15.192:0/463778918 wait complete. 2024-02-20T13:46:11.376 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:11.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:09.373405+0000 mgr.smithi158.gmoobf (mgr.14156) 159 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:12.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:11.386757+0000 mon.smithi158 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.192:0/463778918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:12.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:12.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:13.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:11.373688+0000 mgr.smithi158.gmoobf (mgr.14156) 160 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:14.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- 172.21.15.192:0/1537713611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 msgr2=0x7fe5f00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:14.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 --2- 172.21.15.192:0/1537713611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f00f9160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe5ec009fb0 tx=0x7fe5ec009b30).stop 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- 172.21.15.192:0/1537713611 shutdown_connections 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 --2- 172.21.15.192:0/1537713611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f00f9160 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- 172.21.15.192:0/1537713611 >> 172.21.15.192:0/1537713611 conn(0x7fe5f00f0be0 msgr2=0x7fe5f00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- 172.21.15.192:0/1537713611 shutdown_connections 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- 172.21.15.192:0/1537713611 wait complete. 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 Processor -- start 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- start start 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.890+0000 7fe5f7714700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0106c40 con 0x7fe5f0060360 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f54b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f54b0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56116/0 (socket says 172.21.15.192:56116) 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f54b0700 1 -- 172.21.15.192:0/2468023846 learned_addr learned my addr 172.21.15.192:0/2468023846 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f54b0700 1 -- 172.21.15.192:0/2468023846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec005040 con 0x7fe5f0060360 2024-02-20T13:46:14.910 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f54b0700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe5ec009f80 tx=0x7fe5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec009320 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe5ec013680 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0106e40 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec01eda0 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.891+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f01072d0 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.892+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe5ec01e420 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.892+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0045bb0 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.892+0000 7fe5e5ffb700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 0x7fe5dc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.892+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe5ec0423c0 con 0x7fe5f0060360 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.893+0000 7fe5f4caf700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 0x7fe5dc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.893+0000 7fe5f4caf700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 0x7fe5dc0335a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe5e00096a0 tx=0x7fe5e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:14.897+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe5ec01ccf0 con 0x7fe5f0060360 2024-02-20T13:46:15.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.103+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5f0108180 con 0x7fe5f0060360 2024-02-20T13:46:15.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.103+0000 7fe5e5ffb700 1 -- 172.21.15.192:0/2468023846 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe5ec01cb10 con 0x7fe5f0060360 2024-02-20T13:46:15.091 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:15.091 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:15.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 msgr2=0x7fe5dc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:15.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 0x7fe5dc0335a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe5e00096a0 tx=0x7fe5e0006b40).stop 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 msgr2=0x7fe5f0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe5ec009f80 tx=0x7fe5ec00b040).stop 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 shutdown_connections 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe5dc0310e0 0x7fe5dc0335a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 --2- 172.21.15.192:0/2468023846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5f0060360 0x7fe5f0106750 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 >> 172.21.15.192:0/2468023846 conn(0x7fe5f00f0be0 msgr2=0x7fe5f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 shutdown_connections 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:15.105+0000 7fe5f7714700 1 -- 172.21.15.192:0/2468023846 wait complete. 2024-02-20T13:46:15.094 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:15.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:13.374036+0000 mgr.smithi158.gmoobf (mgr.14156) 161 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:15.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:15.104822+0000 mon.smithi158 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.192:0/2468023846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:16.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:16.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:17.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:15.374370+0000 mgr.smithi158.gmoobf (mgr.14156) 162 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 -- 172.21.15.192:0/3653296503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 msgr2=0x7fa0280f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 --2- 172.21.15.192:0/3653296503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa0280f5030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fa024004770 tx=0x7fa02400ac40).stop 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 -- 172.21.15.192:0/3653296503 shutdown_connections 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 --2- 172.21.15.192:0/3653296503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa0280f5030 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 -- 172.21.15.192:0/3653296503 >> 172.21.15.192:0/3653296503 conn(0x7fa0280f0390 msgr2=0x7fa0280f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 -- 172.21.15.192:0/3653296503 shutdown_connections 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 -- 172.21.15.192:0/3653296503 wait complete. 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.610+0000 7fa02fd42700 1 Processor -- start 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02fd42700 1 -- start start 2024-02-20T13:46:18.599 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02fd42700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02fd42700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281042f0 con 0x7fa0280f4c10 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02dade700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02dade700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56130/0 (socket says 172.21.15.192:56130) 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02dade700 1 -- 172.21.15.192:0/2530724860 learned_addr learned my addr 172.21.15.192:0/2530724860 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02dade700 1 -- 172.21.15.192:0/2530724860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa024005040 con 0x7fa0280f4c10 2024-02-20T13:46:18.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02dade700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fa024004740 tx=0x7fa02400ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:18.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa024019070 con 0x7fa0280f4c10 2024-02-20T13:46:18.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa028104490 con 0x7fa0280f4c10 2024-02-20T13:46:18.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa0240077b0 con 0x7fa0280f4c10 2024-02-20T13:46:18.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.611+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa024008200 con 0x7fa0280f4c10 2024-02-20T13:46:18.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.612+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa028104920 con 0x7fa0280f4c10 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.612+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa0240084e0 con 0x7fa0280f4c10 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.612+0000 7fa01e7fc700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 0x7fa014033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.612+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa010004fa0 con 0x7fa0280f4c10 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.612+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa024041ec0 con 0x7fa0280f4c10 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.613+0000 7fa02d2dd700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 0x7fa014033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:18.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.613+0000 7fa02d2dd700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 0x7fa014033500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:18.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.617+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa02401c500 con 0x7fa0280f4c10 2024-02-20T13:46:18.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.823+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa010005670 con 0x7fa0280f4c10 2024-02-20T13:46:18.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.824+0000 7fa01e7fc700 1 -- 172.21.15.192:0/2530724860 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa02401c320 con 0x7fa0280f4c10 2024-02-20T13:46:18.812 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:18.812 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 msgr2=0x7fa014033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 0x7fa014033500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).stop 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 msgr2=0x7fa028105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fa024004740 tx=0x7fa02400ac40).stop 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 shutdown_connections 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa014031040 0x7fa014033500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 --2- 172.21.15.192:0/2530724860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0280f4c10 0x7fa028105e70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 >> 172.21.15.192:0/2530724860 conn(0x7fa0280f0390 msgr2=0x7fa0280f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:18.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 shutdown_connections 2024-02-20T13:46:18.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:18.825+0000 7fa02fd42700 1 -- 172.21.15.192:0/2530724860 wait complete. 2024-02-20T13:46:18.814 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:19.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:17.374625+0000 mgr.smithi158.gmoobf (mgr.14156) 163 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:19.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:18.825146+0000 mon.smithi158 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.192:0/2530724860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:20.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:20.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:21.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:19.374894+0000 mgr.smithi158.gmoobf (mgr.14156) 164 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.275+0000 7f4a8f2da700 1 -- 172.21.15.192:0/612100496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 msgr2=0x7f4a880f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.275+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/612100496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a880f56f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4a84009fb0 tx=0x7f4a84009b30).stop 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- 172.21.15.192:0/612100496 shutdown_connections 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/612100496 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a880f56f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- 172.21.15.192:0/612100496 >> 172.21.15.192:0/612100496 conn(0x7f4a880f0a30 msgr2=0x7f4a880f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:22.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- 172.21.15.192:0/612100496 shutdown_connections 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- 172.21.15.192:0/612100496 wait complete. 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 Processor -- start 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- start start 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8f2da700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88106af0 con 0x7f4a880f52d0 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8e2d8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8e2d8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56158/0 (socket says 172.21.15.192:56158) 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.276+0000 7f4a8e2d8700 1 -- 172.21.15.192:0/3688932466 learned_addr learned my addr 172.21.15.192:0/3688932466 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a8e2d8700 1 -- 172.21.15.192:0/3688932466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84005040 con 0x7f4a880f52d0 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a8e2d8700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4a84000c00 tx=0x7f4a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a840095f0 con 0x7f4a880f52d0 2024-02-20T13:46:22.266 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a88106cf0 con 0x7f4a880f52d0 2024-02-20T13:46:22.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4a84007e50 con 0x7f4a880f52d0 2024-02-20T13:46:22.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a84014e70 con 0x7f4a880f52d0 2024-02-20T13:46:22.267 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.277+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a88107180 con 0x7f4a880f52d0 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.278+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f4a84025070 con 0x7f4a880f52d0 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.278+0000 7f4a7effd700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 0x7f4a74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.278+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4a84041cf0 con 0x7f4a880f52d0 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.278+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a88045bb0 con 0x7f4a880f52d0 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.278+0000 7f4a8dad7700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 0x7f4a74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:22.268 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.279+0000 7f4a8dad7700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 0x7f4a74033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:22.270 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.283+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a840079c0 con 0x7f4a880f52d0 2024-02-20T13:46:22.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.489+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a880faeb0 con 0x7f4a880f52d0 2024-02-20T13:46:22.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.489+0000 7f4a7effd700 1 -- 172.21.15.192:0/3688932466 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a8400f300 con 0x7f4a880f52d0 2024-02-20T13:46:22.477 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:22.478 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 msgr2=0x7f4a74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 0x7f4a74033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).stop 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 msgr2=0x7f4a88106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4a84000c00 tx=0x7f4a8400b040).stop 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 shutdown_connections 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4a74031090 0x7f4a74033550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 --2- 172.21.15.192:0/3688932466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a880f52d0 0x7f4a88106600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 >> 172.21.15.192:0/3688932466 conn(0x7f4a880f0a30 msgr2=0x7f4a8805fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 shutdown_connections 2024-02-20T13:46:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:22.491+0000 7f4a8f2da700 1 -- 172.21.15.192:0/3688932466 wait complete. 2024-02-20T13:46:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:21.375222+0000 mgr.smithi158.gmoobf (mgr.14156) 165 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:22.490774+0000 mon.smithi158 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.192:0/3688932466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:24.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:24.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:25.504 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:23.375568+0000 mgr.smithi158.gmoobf (mgr.14156) 166 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:25.504 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:24.418571+0000 mon.smithi158 (mon.0) 198 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi158", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T13:46:25.505 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:24.421019+0000 mon.smithi158 (mon.0) 199 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi158", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T13:46:25.505 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:24.421816+0000 mon.smithi158 (mon.0) 200 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:46:25.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 -- 172.21.15.192:0/159801032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 msgr2=0x7fe9b80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:25.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 --2- 172.21.15.192:0/159801032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80f58a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe9a8009fb0 tx=0x7fe9a8009b30).stop 2024-02-20T13:46:25.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 -- 172.21.15.192:0/159801032 shutdown_connections 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 --2- 172.21.15.192:0/159801032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80f58a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 -- 172.21.15.192:0/159801032 >> 172.21.15.192:0/159801032 conn(0x7fe9b80f0be0 msgr2=0x7fe9b80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 -- 172.21.15.192:0/159801032 shutdown_connections 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 -- 172.21.15.192:0/159801032 wait complete. 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.917+0000 7fe9bceca700 1 Processor -- start 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9bceca700 1 -- start start 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9bceca700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9bceca700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b80fb8e0 con 0x7fe9b80f5480 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9b659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:35464/0 (socket says 172.21.15.192:35464) 2024-02-20T13:46:25.907 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9b659c700 1 -- 172.21.15.192:0/3731954813 learned_addr learned my addr 172.21.15.192:0/3731954813 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9b659c700 1 -- 172.21.15.192:0/3731954813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8005040 con 0x7fe9b80f5480 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9b659c700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fe9a80089f0 tx=0x7fe9a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9a8009320 con 0x7fe9b80f5480 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe9a8013ec0 con 0x7fe9b80f5480 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b80fba80 con 0x7fe9b80f5480 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.918+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9a8014d30 con 0x7fe9b80f5480 2024-02-20T13:46:25.908 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.919+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b80fbf10 con 0x7fe9b80f5480 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.919+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe9a80095f0 con 0x7fe9b80f5480 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.919+0000 7fe9a6ffd700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 0x7fe9a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.919+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b80591e0 con 0x7fe9b80f5480 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.919+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe9a8042450 con 0x7fe9b80f5480 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.920+0000 7fe9b5d9b700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 0x7fe9a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:25.909 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.920+0000 7fe9b5d9b700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 0x7fe9a0033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:25.912 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:25.924+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe9a801d7f0 con 0x7fe9b80f5480 2024-02-20T13:46:26.117 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.130+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9b80faf70 con 0x7fe9b80f5480 2024-02-20T13:46:26.118 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.130+0000 7fe9a6ffd700 1 -- 172.21.15.192:0/3731954813 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9a801d7f0 con 0x7fe9b80f5480 2024-02-20T13:46:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:26.119 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:26.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 msgr2=0x7fe9a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:26.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 0x7fe9a0033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).stop 2024-02-20T13:46:26.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 msgr2=0x7fe9b80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:26.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fe9a80089f0 tx=0x7fe9a800b040).stop 2024-02-20T13:46:26.120 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 shutdown_connections 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe9a0031090 0x7fe9a0033550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 --2- 172.21.15.192:0/3731954813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9b80f5480 0x7fe9b80fd460 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 >> 172.21.15.192:0/3731954813 conn(0x7fe9b80f0be0 msgr2=0x7fe9b805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 shutdown_connections 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:26.132+0000 7fe9bceca700 1 -- 172.21.15.192:0/3731954813 wait complete. 2024-02-20T13:46:26.121 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:46:24.416411+0000 mgr.smithi158.gmoobf (mgr.14156) 167 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon alertmanager.smithi158 ... 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:41:48Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:42:25Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:43:03Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:44:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:46:26.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: yield (conn, connr) 2024-02-20T13:46:26.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: code, '\n'.join(err))) 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon alertmanager.smithi158 ... 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:41:48Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:42:25Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:43:03Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:44:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:46:26.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:46:24.416782+0000 mgr.smithi158.gmoobf (mgr.14156) 168 : cephadm [ERR] Failed while placing alertmanager.smithi158 on smithi158: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-alertmanager.smithi158 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon alertmanager.smithi158 ... 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:41:48Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:42:25Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:43:03Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:44:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:45:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:46:26.541 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:46:24.422430+0000 mgr.smithi158.gmoobf (mgr.14156) 169 : cephadm [INF] Deploying daemon crash.smithi158 on smithi158 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:25.457269+0000 mon.smithi158 (mon.0) 201 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:26.542 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.131684+0000 mon.smithi158 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.192:0/3731954813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:27.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:25.375841+0000 mgr.smithi158.gmoobf (mgr.14156) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:27.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.696128+0000 mon.smithi158 (mon.0) 203 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:27.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.839630+0000 mon.smithi158 (mon.0) 204 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.841793+0000 mon.smithi158 (mon.0) 205 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.842170+0000 mon.smithi158 (mon.0) 206 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T13:46:27.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.844687+0000 mon.smithi158 (mon.0) 207 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:27.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:27.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:28.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:26.842495+0000 mgr.smithi158.gmoobf (mgr.14156) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T13:46:28.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:46:26.847642+0000 mgr.smithi158.gmoobf (mgr.14156) 172 : cephadm [INF] Deploying daemon grafana.smithi158 on smithi158 2024-02-20T13:46:29.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:27.376148+0000 mgr.smithi158.gmoobf (mgr.14156) 173 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 -- 172.21.15.192:0/2452491506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 msgr2=0x7fe1e80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/2452491506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e80f69b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fe1dc009fb0 tx=0x7fe1dc009b30).stop 2024-02-20T13:46:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 -- 172.21.15.192:0/2452491506 shutdown_connections 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/2452491506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e80f69b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 -- 172.21.15.192:0/2452491506 >> 172.21.15.192:0/2452491506 conn(0x7fe1e80f1cf0 msgr2=0x7fe1e80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 -- 172.21.15.192:0/2452491506 shutdown_connections 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.795+0000 7fe1ed56c700 1 -- 172.21.15.192:0/2452491506 wait complete. 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.796+0000 7fe1ed56c700 1 Processor -- start 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.796+0000 7fe1ed56c700 1 -- start start 2024-02-20T13:46:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.796+0000 7fe1ed56c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.796+0000 7fe1ed56c700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8106c70 con 0x7fe1e80f6590 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:35476/0 (socket says 172.21.15.192:35476) 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1e6ffd700 1 -- 172.21.15.192:0/4229875166 learned_addr learned my addr 172.21.15.192:0/4229875166 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1e6ffd700 1 -- 172.21.15.192:0/4229875166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc005040 con 0x7fe1e80f6590 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1e6ffd700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fe1dc0089f0 tx=0x7fe1dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1dc009320 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe1dc013ec0 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e8106e70 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1dc014d30 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.797+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e8107300 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.798+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe1dc014420 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.798+0000 7fe1cf7fe700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 0x7fe1d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.798+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe1dc042390 con 0x7fe1e80f6590 2024-02-20T13:46:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.798+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8045bb0 con 0x7fe1e80f6590 2024-02-20T13:46:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.799+0000 7fe1e67fc700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 0x7fe1d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.799+0000 7fe1e67fc700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 0x7fe1d0033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe1d80096a0 tx=0x7fe1d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:29.803+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe1dc014750 con 0x7fe1e80f6590 2024-02-20T13:46:29.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.008+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1e8107f80 con 0x7fe1e80f6590 2024-02-20T13:46:29.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.010+0000 7fe1cf7fe700 1 -- 172.21.15.192:0/4229875166 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1dc01d7f0 con 0x7fe1e80f6590 2024-02-20T13:46:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:29.997 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:29.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 msgr2=0x7fe1d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 0x7fe1d0033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe1d80096a0 tx=0x7fe1d8006b40).stop 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 msgr2=0x7fe1e8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fe1dc0089f0 tx=0x7fe1dc00b040).stop 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 shutdown_connections 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe1d0031090 0x7fe1d0033550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 --2- 172.21.15.192:0/4229875166 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1e80f6590 0x7fe1e8106780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 >> 172.21.15.192:0/4229875166 conn(0x7fe1e80f1cf0 msgr2=0x7fe1e80f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:30.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 shutdown_connections 2024-02-20T13:46:30.000 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:30.011+0000 7fe1ed56c700 1 -- 172.21.15.192:0/4229875166 wait complete. 2024-02-20T13:46:30.000 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:30.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:30.010330+0000 mon.smithi158 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.192:0/4229875166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:31.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:29.376537+0000 mgr.smithi158.gmoobf (mgr.14156) 174 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:31.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:30.460201+0000 mon.smithi158 (mon.0) 209 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:31.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:31.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:33.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:31.376922+0000 mgr.smithi158.gmoobf (mgr.14156) 175 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:33.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.582+0000 7f21a3fff700 1 -- 172.21.15.192:0/3023222434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 msgr2=0x7f21a40f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:33.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.582+0000 7f21a3fff700 1 --2- 172.21.15.192:0/3023222434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a40f6760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f2194009fb0 tx=0x7f2194009b30).stop 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- 172.21.15.192:0/3023222434 shutdown_connections 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 --2- 172.21.15.192:0/3023222434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a40f6760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- 172.21.15.192:0/3023222434 >> 172.21.15.192:0/3023222434 conn(0x7f21a40f1aa0 msgr2=0x7f21a40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- 172.21.15.192:0/3023222434 shutdown_connections 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- 172.21.15.192:0/3023222434 wait complete. 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 Processor -- start 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- start start 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.583+0000 7f21a3fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4108c80 con 0x7f21a40f6340 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53878/0 (socket says 172.21.15.192:53878) 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a2ffd700 1 -- 172.21.15.192:0/4291024594 learned_addr learned my addr 172.21.15.192:0/4291024594 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a2ffd700 1 -- 172.21.15.192:0/4291024594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194005040 con 0x7f21a40f6340 2024-02-20T13:46:33.573 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a2ffd700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f21940089f0 tx=0x7f219400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2194009320 con 0x7f21a40f6340 2024-02-20T13:46:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a4108e80 con 0x7f21a40f6340 2024-02-20T13:46:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2194013ec0 con 0x7f21a40f6340 2024-02-20T13:46:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2194014d30 con 0x7f21a40f6340 2024-02-20T13:46:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.584+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a4109310 con 0x7f21a40f6340 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.585+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f21940095f0 con 0x7f21a40f6340 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.585+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a4045bb0 con 0x7f21a40f6340 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.585+0000 7f218b7fe700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 0x7f218c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.585+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2194042390 con 0x7f21a40f6340 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.586+0000 7f21a27fc700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 0x7f218c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:33.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.586+0000 7f21a27fc700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 0x7f218c033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f21980096a0 tx=0x7f2198006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:33.577 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.590+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f219401d7f0 con 0x7f21a40f6340 2024-02-20T13:46:33.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.793+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21a406b400 con 0x7f21a40f6340 2024-02-20T13:46:33.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.794+0000 7f218b7fe700 1 -- 172.21.15.192:0/4291024594 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f219401d7f0 con 0x7f21a40f6340 2024-02-20T13:46:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:33.782 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:33.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 msgr2=0x7f218c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:33.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 0x7f218c033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f21980096a0 tx=0x7f2198006b40).stop 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 msgr2=0x7f21a4108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f21940089f0 tx=0x7f219400b040).stop 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 shutdown_connections 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f218c031090 0x7f218c033550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 --2- 172.21.15.192:0/4291024594 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21a40f6340 0x7f21a4108790 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 >> 172.21.15.192:0/4291024594 conn(0x7f21a40f1aa0 msgr2=0x7f21a40f3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 shutdown_connections 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:33.796+0000 7f21a3fff700 1 -- 172.21.15.192:0/4291024594 wait complete. 2024-02-20T13:46:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:34.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:33.795596+0000 mon.smithi158 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.192:0/4291024594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:35.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:33.377241+0000 mgr.smithi158.gmoobf (mgr.14156) 176 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:35.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:35.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/1164585467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 msgr2=0x7fcdf40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/1164585467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf40f58a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fcde4009fb0 tx=0x7fcde4009b30).stop 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/1164585467 shutdown_connections 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/1164585467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf40f58a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/1164585467 >> 172.21.15.192:0/1164585467 conn(0x7fcdf40f0be0 msgr2=0x7fcdf40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/1164585467 shutdown_connections 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/1164585467 wait complete. 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 Processor -- start 2024-02-20T13:46:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.520+0000 7fcdfbbfa700 1 -- start start 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdfbbfa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdfbbfa700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4106cc0 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdf9996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdf9996700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53892/0 (socket says 172.21.15.192:53892) 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdf9996700 1 -- 172.21.15.192:0/3087466720 learned_addr learned my addr 172.21.15.192:0/3087466720 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdf9996700 1 -- 172.21.15.192:0/3087466720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4005040 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdf9996700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fcde40089f0 tx=0x7fcde400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde4009320 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcde4013ec0 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf4106ec0 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde4014d30 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.521+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf4107350 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.522+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fcde40263a0 con 0x7fcdf40f5480 2024-02-20T13:46:37.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.522+0000 7fcdea7fc700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 0x7fcde00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:37.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.522+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcde4042430 con 0x7fcdf40f5480 2024-02-20T13:46:37.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.522+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf4045bb0 con 0x7fcdf40f5480 2024-02-20T13:46:37.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.522+0000 7fcdf9195700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 0x7fcde00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:37.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.523+0000 7fcdf9195700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 0x7fcde00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fcdf00096a0 tx=0x7fcdf0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:37.514 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.528+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcde4012920 con 0x7fcdf40f5480 2024-02-20T13:46:37.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:35.377482+0000 mgr.smithi158.gmoobf (mgr.14156) 177 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:37.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.732+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdf41080a0 con 0x7fcdf40f5480 2024-02-20T13:46:37.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.732+0000 7fcdea7fc700 1 -- 172.21.15.192:0/3087466720 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcde402b020 con 0x7fcdf40f5480 2024-02-20T13:46:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:37.720 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 msgr2=0x7fcde00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 0x7fcde00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fcdf00096a0 tx=0x7fcdf0006b40).stop 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 msgr2=0x7fcdf41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fcde40089f0 tx=0x7fcde400b040).stop 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 shutdown_connections 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcde00310e0 0x7fcde00335a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 --2- 172.21.15.192:0/3087466720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdf40f5480 0x7fcdf41067d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 >> 172.21.15.192:0/3087466720 conn(0x7fcdf40f0be0 msgr2=0x7fcdf405fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 shutdown_connections 2024-02-20T13:46:37.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:37.734+0000 7fcdfbbfa700 1 -- 172.21.15.192:0/3087466720 wait complete. 2024-02-20T13:46:37.722 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:38.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:37.732895+0000 mon.smithi158 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.192:0/3087466720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:39.439 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:39.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:39.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:37.377749+0000 mgr.smithi158.gmoobf (mgr.14156) 178 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:41.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 -- 172.21.15.192:0/2915599766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 msgr2=0x7f6a200f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:41.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 --2- 172.21.15.192:0/2915599766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a200f69b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f6a10004760 tx=0x7f6a10009ad0).stop 2024-02-20T13:46:41.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 -- 172.21.15.192:0/2915599766 shutdown_connections 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 --2- 172.21.15.192:0/2915599766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a200f69b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 -- 172.21.15.192:0/2915599766 >> 172.21.15.192:0/2915599766 conn(0x7f6a200f1cd0 msgr2=0x7f6a200f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 -- 172.21.15.192:0/2915599766 shutdown_connections 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 -- 172.21.15.192:0/2915599766 wait complete. 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.424+0000 7f6a285cb700 1 Processor -- start 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a285cb700 1 -- start start 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a285cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a285cb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20106cb0 con 0x7f6a200f6590 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a26367700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a26367700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53898/0 (socket says 172.21.15.192:53898) 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a26367700 1 -- 172.21.15.192:0/99990178 learned_addr learned my addr 172.21.15.192:0/99990178 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a26367700 1 -- 172.21.15.192:0/99990178 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a10009d70 con 0x7f6a200f6590 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a26367700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6a10004170 tx=0x7f6a10009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a1000e040 con 0x7f6a200f6590 2024-02-20T13:46:41.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6a10008ef0 con 0x7f6a200f6590 2024-02-20T13:46:41.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a20106eb0 con 0x7f6a200f6590 2024-02-20T13:46:41.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a10014c90 con 0x7f6a200f6590 2024-02-20T13:46:41.414 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.425+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a20107340 con 0x7f6a200f6590 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.426+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6a100283f0 con 0x7f6a200f6590 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.426+0000 7f6a16ffd700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 0x7f6a0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.426+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6a10042200 con 0x7f6a200f6590 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.426+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20045bb0 con 0x7f6a200f6590 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.426+0000 7f6a25b66700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 0x7f6a0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:41.415 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.427+0000 7f6a25b66700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 0x7f6a0c033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6a1c0096a0 tx=0x7f6a1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:41.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.431+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6a100268f0 con 0x7f6a200f6590 2024-02-20T13:46:41.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:39.378051+0000 mgr.smithi158.gmoobf (mgr.14156) 179 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:41.622 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.635+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a20107fc0 con 0x7f6a200f6590 2024-02-20T13:46:41.623 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.635+0000 7f6a16ffd700 1 -- 172.21.15.192:0/99990178 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a10026300 con 0x7f6a200f6590 2024-02-20T13:46:41.623 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:41.623 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:41.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 msgr2=0x7f6a0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:41.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 0x7f6a0c033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6a1c0096a0 tx=0x7f6a1c006b40).stop 2024-02-20T13:46:41.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 msgr2=0x7f6a201067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:41.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6a10004170 tx=0x7f6a10009ad0).stop 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 shutdown_connections 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6a0c031090 0x7f6a0c033550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 --2- 172.21.15.192:0/99990178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a200f6590 0x7f6a201067c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 >> 172.21.15.192:0/99990178 conn(0x7f6a200f1cd0 msgr2=0x7f6a200f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 shutdown_connections 2024-02-20T13:46:41.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:41.637+0000 7f6a285cb700 1 -- 172.21.15.192:0/99990178 wait complete. 2024-02-20T13:46:41.627 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:42.330 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:41.635985+0000 mon.smithi158 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.192:0/99990178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:43.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:43.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:43.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:41.378305+0000 mgr.smithi158.gmoobf (mgr.14156) 180 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:45.299 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 -- 172.21.15.192:0/3508545011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 msgr2=0x7f2f6c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:45.299 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 --2- 172.21.15.192:0/3508545011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c0f69b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f5c009fb0 tx=0x7f2f5c009b30).stop 2024-02-20T13:46:45.299 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 -- 172.21.15.192:0/3508545011 shutdown_connections 2024-02-20T13:46:45.299 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 --2- 172.21.15.192:0/3508545011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c0f69b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 -- 172.21.15.192:0/3508545011 >> 172.21.15.192:0/3508545011 conn(0x7f2f6c0f1cf0 msgr2=0x7f2f6c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 -- 172.21.15.192:0/3508545011 shutdown_connections 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.310+0000 7f2f719d0700 1 -- 172.21.15.192:0/3508545011 wait complete. 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.311+0000 7f2f719d0700 1 Processor -- start 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.311+0000 7f2f719d0700 1 -- start start 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.311+0000 7f2f719d0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.311+0000 7f2f719d0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c106c70 con 0x7f2f6c0f6590 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f6affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f6affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:57584/0 (socket says 172.21.15.192:57584) 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f6affd700 1 -- 172.21.15.192:0/2540890080 learned_addr learned my addr 172.21.15.192:0/2540890080 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:45.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f6affd700 1 -- 172.21.15.192:0/2540890080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c005040 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f6affd700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2f5c0089f0 tx=0x7f2f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c009320 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c106e70 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2f5c013ec0 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c014d30 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.312+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c107300 con 0x7f2f6c0f6590 2024-02-20T13:46:45.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2f5c0263a0 con 0x7f2f6c0f6590 2024-02-20T13:46:45.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f537fe700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 0x7f2f54033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:45.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2f5c042380 con 0x7f2f6c0f6590 2024-02-20T13:46:45.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c045bb0 con 0x7f2f6c0f6590 2024-02-20T13:46:45.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f6a7fc700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 0x7f2f54033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:45.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.313+0000 7f2f6a7fc700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 0x7f2f54033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2f600096a0 tx=0x7f2f60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:45.304 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.318+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2f5c0266d0 con 0x7f2f6c0f6590 2024-02-20T13:46:45.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.522+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f6c108040 con 0x7f2f6c0f6590 2024-02-20T13:46:45.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.523+0000 7f2f537fe700 1 -- 172.21.15.192:0/2540890080 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f5c02b020 con 0x7f2f6c0f6590 2024-02-20T13:46:45.509 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:45.510 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 msgr2=0x7f2f54033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 0x7f2f54033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2f600096a0 tx=0x7f2f60006b40).stop 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 msgr2=0x7f2f6c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2f5c0089f0 tx=0x7f2f5c00b040).stop 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 shutdown_connections 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2f54031090 0x7f2f54033550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 --2- 172.21.15.192:0/2540890080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f6c0f6590 0x7f2f6c106780 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:45.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 >> 172.21.15.192:0/2540890080 conn(0x7f2f6c0f1cf0 msgr2=0x7f2f6c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:45.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 shutdown_connections 2024-02-20T13:46:45.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:45.524+0000 7f2f719d0700 1 -- 172.21.15.192:0/2540890080 wait complete. 2024-02-20T13:46:45.512 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:45.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:43.378619+0000 mgr.smithi158.gmoobf (mgr.14156) 181 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:46.783 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:45.310147+0000 mon.smithi158 (mon.0) 213 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:46.783 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:46:45.319021+0000 mgr.smithi158.gmoobf (mgr.14156) 182 : cephadm [INF] Deploying daemon node-exporter.smithi158 on smithi158 2024-02-20T13:46:46.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:45.378905+0000 mgr.smithi158.gmoobf (mgr.14156) 183 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:46.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:45.463255+0000 mon.smithi158 (mon.0) 214 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:46:46.784 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:45.523141+0000 mon.smithi158 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.192:0/2540890080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:47.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:47.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:48.783 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:47.379246+0000 mgr.smithi158.gmoobf (mgr.14156) 184 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 -- 172.21.15.192:0/3110776633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 msgr2=0x7ff7040f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3110776633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7040f69b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff6f8004760 tx=0x7ff6f8009ad0).stop 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 -- 172.21.15.192:0/3110776633 shutdown_connections 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3110776633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7040f69b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 -- 172.21.15.192:0/3110776633 >> 172.21.15.192:0/3110776633 conn(0x7ff7040f1cd0 msgr2=0x7ff7040f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 -- 172.21.15.192:0/3110776633 shutdown_connections 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.989+0000 7ff7090b3700 1 -- 172.21.15.192:0/3110776633 wait complete. 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff7090b3700 1 Processor -- start 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff7090b3700 1 -- start start 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff7090b3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff7090b3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704106cb0 con 0x7ff7040f6590 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff702d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff702d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:57602/0 (socket says 172.21.15.192:57602) 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff702d9d700 1 -- 172.21.15.192:0/3791400371 learned_addr learned my addr 172.21.15.192:0/3791400371 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff702d9d700 1 -- 172.21.15.192:0/3791400371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f8009d70 con 0x7ff7040f6590 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.990+0000 7ff702d9d700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff6f8004170 tx=0x7ff6f8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f800e040 con 0x7ff7040f6590 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff704106eb0 con 0x7ff7040f6590 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff6f8008ef0 con 0x7ff7040f6590 2024-02-20T13:46:48.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f8014c90 con 0x7ff7040f6590 2024-02-20T13:46:48.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff704107340 con 0x7ff7040f6590 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.991+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff6f80283f0 con 0x7ff7040f6590 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.992+0000 7ff6eb7fe700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 0x7ff6ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.992+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff6f80421d0 con 0x7ff7040f6590 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.992+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7040591e0 con 0x7ff7040f6590 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.992+0000 7ff70259c700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 0x7ff6ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:48.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.992+0000 7ff70259c700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 0x7ff6ec033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff6f40096a0 tx=0x7ff6f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:48.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:48.996+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff6f80268f0 con 0x7ff7040f6590 2024-02-20T13:46:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.203+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff704045bb0 con 0x7ff7040f6590 2024-02-20T13:46:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.204+0000 7ff6eb7fe700 1 -- 172.21.15.192:0/3791400371 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff6f8026300 con 0x7ff7040f6590 2024-02-20T13:46:49.191 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:49.191 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:49.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 msgr2=0x7ff6ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:49.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 0x7ff6ec033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff6f40096a0 tx=0x7ff6f4006b40).stop 2024-02-20T13:46:49.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 msgr2=0x7ff7041067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:49.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff6f8004170 tx=0x7ff6f8009ad0).stop 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 shutdown_connections 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff6ec031090 0x7ff6ec033550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 --2- 172.21.15.192:0/3791400371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff7040f6590 0x7ff7041067c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.205+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 >> 172.21.15.192:0/3791400371 conn(0x7ff7040f1cd0 msgr2=0x7ff7040f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.206+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 shutdown_connections 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:49.206+0000 7ff7090b3700 1 -- 172.21.15.192:0/3791400371 wait complete. 2024-02-20T13:46:49.193 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:50.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:49.204276+0000 mon.smithi158 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.192:0/3791400371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:50.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:50.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:51.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:49.379499+0000 mgr.smithi158.gmoobf (mgr.14156) 185 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:52.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 -- 172.21.15.192:0/3801068194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 msgr2=0x7f21140f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 --2- 172.21.15.192:0/3801068194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f21140f9160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f2108009fb0 tx=0x7f2108009b30).stop 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 -- 172.21.15.192:0/3801068194 shutdown_connections 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 --2- 172.21.15.192:0/3801068194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f21140f9160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 -- 172.21.15.192:0/3801068194 >> 172.21.15.192:0/3801068194 conn(0x7f21140f0be0 msgr2=0x7f21140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 -- 172.21.15.192:0/3801068194 shutdown_connections 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 -- 172.21.15.192:0/3801068194 wait complete. 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.730+0000 7f2119b10700 1 Processor -- start 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f2119b10700 1 -- start start 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f2119b10700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f2119b10700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2114106c40 con 0x7f2114060360 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f21137fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f21137fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44664/0 (socket says 172.21.15.192:44664) 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f21137fe700 1 -- 172.21.15.192:0/693589794 learned_addr learned my addr 172.21.15.192:0/693589794 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f21137fe700 1 -- 172.21.15.192:0/693589794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108005040 con 0x7f2114060360 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.731+0000 7f21137fe700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f2108009f80 tx=0x7f210800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2108009320 con 0x7f2114060360 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2114106e40 con 0x7f2114060360 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2108013680 con 0x7f2114060360 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f210801eda0 con 0x7f2114060360 2024-02-20T13:46:52.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21141072d0 con 0x7f2114060360 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.732+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f210801e420 con 0x7f2114060360 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.733+0000 7f20fbfff700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 0x7f20fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.733+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2114045bb0 con 0x7f2114060360 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.733+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f21080423e0 con 0x7f2114060360 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.733+0000 7f2112ffd700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 0x7f20fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:52.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.733+0000 7f2112ffd700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 0x7f20fc033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f21040096a0 tx=0x7f2104006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:52.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.737+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f210801d380 con 0x7f2114060360 2024-02-20T13:46:52.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.944+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2114108120 con 0x7f2114060360 2024-02-20T13:46:52.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.944+0000 7f20fbfff700 1 -- 172.21.15.192:0/693589794 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2108004490 con 0x7f2114060360 2024-02-20T13:46:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:52.932 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:52.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 msgr2=0x7f20fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:52.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 0x7f20fc033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f21040096a0 tx=0x7f2104006b40).stop 2024-02-20T13:46:52.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 msgr2=0x7f2114106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:52.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f2108009f80 tx=0x7f210800b040).stop 2024-02-20T13:46:52.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 shutdown_connections 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f20fc031090 0x7f20fc033550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 --2- 172.21.15.192:0/693589794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2114060360 0x7f2114106750 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 >> 172.21.15.192:0/693589794 conn(0x7f21140f0be0 msgr2=0x7f21140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 shutdown_connections 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:52.946+0000 7f2119b10700 1 -- 172.21.15.192:0/693589794 wait complete. 2024-02-20T13:46:52.934 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:53.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:51.379820+0000 mgr.smithi158.gmoobf (mgr.14156) 186 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:53.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:52.945618+0000 mon.smithi158 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.192:0/693589794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:54.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:54.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:55.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:53.380173+0000 mgr.smithi158.gmoobf (mgr.14156) 187 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:56.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- 172.21.15.192:0/3844874277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 msgr2=0x7f89f40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:56.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/3844874277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f40f56f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f89f0009fb0 tx=0x7f89f0009b30).stop 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- 172.21.15.192:0/3844874277 shutdown_connections 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/3844874277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f40f56f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- 172.21.15.192:0/3844874277 >> 172.21.15.192:0/3844874277 conn(0x7f89f40f0a30 msgr2=0x7f89f40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- 172.21.15.192:0/3844874277 shutdown_connections 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- 172.21.15.192:0/3844874277 wait complete. 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 Processor -- start 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- start start 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.504+0000 7f89fa4d9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4060320 con 0x7f89f40f52d0 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89f94d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89f94d7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44686/0 (socket says 172.21.15.192:44686) 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89f94d7700 1 -- 172.21.15.192:0/1251431457 learned_addr learned my addr 172.21.15.192:0/1251431457 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89f94d7700 1 -- 172.21.15.192:0/1251431457 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f0005040 con 0x7f89f40f52d0 2024-02-20T13:46:56.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89f94d7700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f89f0000c00 tx=0x7f89f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f00095f0 con 0x7f89f40f52d0 2024-02-20T13:46:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f4060520 con 0x7f89f40f52d0 2024-02-20T13:46:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f89f0007e50 con 0x7f89f40f52d0 2024-02-20T13:46:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.505+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f0014e70 con 0x7f89f40f52d0 2024-02-20T13:46:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.506+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f41099c0 con 0x7f89f40f52d0 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.506+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f89f0025070 con 0x7f89f40f52d0 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.507+0000 7f89e9ffb700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 0x7f89e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.507+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f4045bb0 con 0x7f89f40f52d0 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.507+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f89f0041d10 con 0x7f89f40f52d0 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.507+0000 7f89f8cd6700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 0x7f89e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:46:56.496 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.507+0000 7f89f8cd6700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 0x7f89e0033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f89e40096a0 tx=0x7f89e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:46:56.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.512+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f89f00079c0 con 0x7f89f40f52d0 2024-02-20T13:46:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:55.464018+0000 mon.smithi158 (mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:46:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:55.464286+0000 mon.smithi158 (mon.0) 219 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:46:56.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.716+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89f410a1b0 con 0x7f89f40f52d0 2024-02-20T13:46:56.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.717+0000 7f89e9ffb700 1 -- 172.21.15.192:0/1251431457 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89f000f300 con 0x7f89f40f52d0 2024-02-20T13:46:56.705 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:46:56.705 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 msgr2=0x7f89e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 0x7f89e0033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f89e40096a0 tx=0x7f89e4006b40).stop 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 msgr2=0x7f89f4109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f89f0000c00 tx=0x7f89f000b040).stop 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 shutdown_connections 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f89e0031090 0x7f89e0033550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 --2- 172.21.15.192:0/1251431457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40f52d0 0x7f89f4109310 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 >> 172.21.15.192:0/1251431457 conn(0x7f89f40f0a30 msgr2=0x7f89f40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 shutdown_connections 2024-02-20T13:46:56.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:46:56.718+0000 7f89fa4d9700 1 -- 172.21.15.192:0/1251431457 wait complete. 2024-02-20T13:46:56.707 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:46:57.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:55.380466+0000 mgr.smithi158.gmoobf (mgr.14156) 188 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:46:57.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:46:56.718068+0000 mon.smithi158 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.192:0/1251431457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:46:58.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:46:58.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:46:59.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:46:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:57.380766+0000 mgr.smithi158.gmoobf (mgr.14156) 189 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:00.458 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 -- 172.21.15.192:0/2680589201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 msgr2=0x7f163c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 --2- 172.21.15.192:0/2680589201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0f58d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f1630009fb0 tx=0x7f1630009b30).stop 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 -- 172.21.15.192:0/2680589201 shutdown_connections 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 --2- 172.21.15.192:0/2680589201 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0f58d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 -- 172.21.15.192:0/2680589201 >> 172.21.15.192:0/2680589201 conn(0x7f163c0f0c10 msgr2=0x7f163c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 -- 172.21.15.192:0/2680589201 shutdown_connections 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.469+0000 7f1643d4e700 1 -- 172.21.15.192:0/2680589201 wait complete. 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1643d4e700 1 Processor -- start 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1643d4e700 1 -- start start 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1643d4e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1643d4e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0fb910 con 0x7f163c0f54b0 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1641aea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:00.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1641aea700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44708/0 (socket says 172.21.15.192:44708) 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.470+0000 7f1641aea700 1 -- 172.21.15.192:0/3494633905 learned_addr learned my addr 172.21.15.192:0/3494633905 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f1641aea700 1 -- 172.21.15.192:0/3494633905 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630005040 con 0x7f163c0f54b0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f1641aea700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630009320 con 0x7f163c0f54b0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c0fbab0 con 0x7f163c0f54b0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1630013ec0 con 0x7f163c0f54b0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630014d30 con 0x7f163c0f54b0 2024-02-20T13:47:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.471+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c0fbf40 con 0x7f163c0f54b0 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.472+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1630014420 con 0x7f163c0f54b0 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.472+0000 7f162e7fc700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 0x7f1628033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.472+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c0591e0 con 0x7f163c0f54b0 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.472+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1630042390 con 0x7f163c0f54b0 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.472+0000 7f16412e9700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 0x7f1628033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.473+0000 7f16412e9700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 0x7f1628033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f16380096a0 tx=0x7f1638006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:00.465 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.477+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f163001d7f0 con 0x7f163c0f54b0 2024-02-20T13:47:00.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.685+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f163c0fc6e0 con 0x7f163c0f54b0 2024-02-20T13:47:00.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.686+0000 7f162e7fc700 1 -- 172.21.15.192:0/3494633905 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f163001d7f0 con 0x7f163c0f54b0 2024-02-20T13:47:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:00.674 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:00.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 msgr2=0x7f1628033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:00.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 0x7f1628033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f16380096a0 tx=0x7f1638006b40).stop 2024-02-20T13:47:00.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 msgr2=0x7f163c0fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).stop 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 shutdown_connections 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1628031090 0x7f1628033550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 --2- 172.21.15.192:0/3494633905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f163c0f54b0 0x7f163c0fd490 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 >> 172.21.15.192:0/3494633905 conn(0x7f163c0f0c10 msgr2=0x7f163c05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 shutdown_connections 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:00.688+0000 7f1643d4e700 1 -- 172.21.15.192:0/3494633905 wait complete. 2024-02-20T13:47:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:01.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:46:59.381089+0000 mgr.smithi158.gmoobf (mgr.14156) 190 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:01.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:00.687232+0000 mon.smithi158 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.192:0/3494633905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:02.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:02.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:03.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:01.381377+0000 mgr.smithi158.gmoobf (mgr.14156) 191 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:04.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 -- 172.21.15.192:0/3333740766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 msgr2=0x7fd0140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 --2- 172.21.15.192:0/3333740766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0140f58a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd004009fb0 tx=0x7fd004009b30).stop 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 -- 172.21.15.192:0/3333740766 shutdown_connections 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 --2- 172.21.15.192:0/3333740766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0140f58a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 -- 172.21.15.192:0/3333740766 >> 172.21.15.192:0/3333740766 conn(0x7fd0140f0be0 msgr2=0x7fd0140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 -- 172.21.15.192:0/3333740766 shutdown_connections 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.144+0000 7fd01a727700 1 -- 172.21.15.192:0/3333740766 wait complete. 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.145+0000 7fd01a727700 1 Processor -- start 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.145+0000 7fd01a727700 1 -- start start 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd01a727700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd01a727700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014060320 con 0x7fd0140f5480 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd013fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd013fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58870/0 (socket says 172.21.15.192:58870) 2024-02-20T13:47:04.137 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd013fff700 1 -- 172.21.15.192:0/3332566409 learned_addr learned my addr 172.21.15.192:0/3332566409 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd013fff700 1 -- 172.21.15.192:0/3332566409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004005040 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd013fff700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fd0040089f0 tx=0x7fd00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004009320 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd014060520 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd004013ec0 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004014d30 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.146+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd014109ba0 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.147+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fd004014420 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.147+0000 7fd010ff9700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 0x7fcffc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.147+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014045bb0 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.147+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd0040424e0 con 0x7fd0140f5480 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.148+0000 7fd0137fe700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 0x7fcffc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.148+0000 7fd0137fe700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 0x7fcffc033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd0080096a0 tx=0x7fd008006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:04.140 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.153+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd00401d7f0 con 0x7fd0140f5480 2024-02-20T13:47:04.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.358+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0140fb060 con 0x7fd0140f5480 2024-02-20T13:47:04.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.359+0000 7fd010ff9700 1 -- 172.21.15.192:0/3332566409 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd00401b2d0 con 0x7fd0140f5480 2024-02-20T13:47:04.346 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:04.346 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 msgr2=0x7fcffc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 0x7fcffc033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd0080096a0 tx=0x7fd008006b40).stop 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 msgr2=0x7fd0141094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fd0040089f0 tx=0x7fd00400b040).stop 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 shutdown_connections 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcffc031090 0x7fcffc033550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 --2- 172.21.15.192:0/3332566409 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0140f5480 0x7fd0141094f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 >> 172.21.15.192:0/3332566409 conn(0x7fd0140f0be0 msgr2=0x7fd0140f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 shutdown_connections 2024-02-20T13:47:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:04.360+0000 7fd01a727700 1 -- 172.21.15.192:0/3332566409 wait complete. 2024-02-20T13:47:04.349 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:05.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:03.381673+0000 mgr.smithi158.gmoobf (mgr.14156) 192 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:05.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:04.360039+0000 mon.smithi158 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.192:0/3332566409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:05.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:05.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:07.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:05.381936+0000 mgr.smithi158.gmoobf (mgr.14156) 193 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:07.973 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 -- 172.21.15.192:0/1989902536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 msgr2=0x7fbb240f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:07.973 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 --2- 172.21.15.192:0/1989902536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb240f5710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fbb18009fb0 tx=0x7fbb18009b30).stop 2024-02-20T13:47:07.973 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 -- 172.21.15.192:0/1989902536 shutdown_connections 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 --2- 172.21.15.192:0/1989902536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb240f5710 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 -- 172.21.15.192:0/1989902536 >> 172.21.15.192:0/1989902536 conn(0x7fbb240f0a70 msgr2=0x7fbb240f2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 -- 172.21.15.192:0/1989902536 shutdown_connections 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.981+0000 7fbb29803700 1 -- 172.21.15.192:0/1989902536 wait complete. 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb29803700 1 Processor -- start 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb29803700 1 -- start start 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb29803700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb29803700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24105a60 con 0x7fbb240f52f0 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb22ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb22ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58896/0 (socket says 172.21.15.192:58896) 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.982+0000 7fbb22ffd700 1 -- 172.21.15.192:0/1206362796 learned_addr learned my addr 172.21.15.192:0/1206362796 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb22ffd700 1 -- 172.21.15.192:0/1206362796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb18005040 con 0x7fbb240f52f0 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb22ffd700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fbb180089f0 tx=0x7fbb1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb18009320 con 0x7fbb240f52f0 2024-02-20T13:47:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb24105c60 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbb18013ec0 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb18014d30 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.983+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb241060f0 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fbb180263a0 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb0b7fe700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 0x7fbb0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbb180423d0 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb240591e0 con 0x7fbb240f52f0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb227fc700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 0x7fbb0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.984+0000 7fbb227fc700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 0x7fbb0c033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbb140096a0 tx=0x7fbb14006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:07.989+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbb180266d0 con 0x7fbb240f52f0 2024-02-20T13:47:08.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.197+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb240fade0 con 0x7fbb240f52f0 2024-02-20T13:47:08.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.197+0000 7fbb0b7fe700 1 -- 172.21.15.192:0/1206362796 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb1802b020 con 0x7fbb240f52f0 2024-02-20T13:47:08.185 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:08.185 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:08.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 msgr2=0x7fbb0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:08.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 0x7fbb0c033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbb140096a0 tx=0x7fbb14006b40).stop 2024-02-20T13:47:08.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 msgr2=0x7fbb24105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:08.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fbb180089f0 tx=0x7fbb1800b040).stop 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 shutdown_connections 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbb0c031090 0x7fbb0c033550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 --2- 172.21.15.192:0/1206362796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb240f52f0 0x7fbb24105570 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 >> 172.21.15.192:0/1206362796 conn(0x7fbb240f0a70 msgr2=0x7fbb240f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 shutdown_connections 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:08.199+0000 7fbb29803700 1 -- 172.21.15.192:0/1206362796 wait complete. 2024-02-20T13:47:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:09.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:07.382299+0000 mgr.smithi158.gmoobf (mgr.14156) 194 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:09.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:08.198686+0000 mon.smithi158 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.192:0/1206362796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:09.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:09.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:11.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:09.382607+0000 mgr.smithi158.gmoobf (mgr.14156) 195 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:11.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 -- 172.21.15.192:0/4264526604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 msgr2=0x7fda280f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:11.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/4264526604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda280f58a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fda10009fb0 tx=0x7fda10009b30).stop 2024-02-20T13:47:11.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 -- 172.21.15.192:0/4264526604 shutdown_connections 2024-02-20T13:47:11.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/4264526604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda280f58a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:11.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 -- 172.21.15.192:0/4264526604 >> 172.21.15.192:0/4264526604 conn(0x7fda280f0be0 msgr2=0x7fda280f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:11.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 -- 172.21.15.192:0/4264526604 shutdown_connections 2024-02-20T13:47:11.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.766+0000 7fda2d0f1700 1 -- 172.21.15.192:0/4264526604 wait complete. 2024-02-20T13:47:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda2d0f1700 1 Processor -- start 2024-02-20T13:47:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda2d0f1700 1 -- start start 2024-02-20T13:47:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda2d0f1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda2d0f1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28105be0 con 0x7fda280f5480 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda26d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.767+0000 7fda26d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58918/0 (socket says 172.21.15.192:58918) 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda26d9d700 1 -- 172.21.15.192:0/1293407037 learned_addr learned my addr 172.21.15.192:0/1293407037 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda26d9d700 1 -- 172.21.15.192:0/1293407037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda10005040 con 0x7fda280f5480 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda26d9d700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fda100089f0 tx=0x7fda1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:11.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda10009320 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28105de0 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fda10013ec0 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda10014d30 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.768+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda28106270 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.769+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fda10014420 con 0x7fda280f5480 2024-02-20T13:47:11.761 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.769+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda28045bb0 con 0x7fda280f5480 2024-02-20T13:47:11.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.769+0000 7fda1f7fe700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 0x7fda14033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:11.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.769+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fda100441a0 con 0x7fda280f5480 2024-02-20T13:47:11.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.770+0000 7fda2659c700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 0x7fda14033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:11.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.770+0000 7fda2659c700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 0x7fda14033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fda180096a0 tx=0x7fda18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:11.763 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.774+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fda1001f980 con 0x7fda280f5480 2024-02-20T13:47:11.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.980+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda280faf70 con 0x7fda280f5480 2024-02-20T13:47:11.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.980+0000 7fda1f7fe700 1 -- 172.21.15.192:0/1293407037 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda10027e80 con 0x7fda280f5480 2024-02-20T13:47:11.968 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:11.968 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:11.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 msgr2=0x7fda14033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:11.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 0x7fda14033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fda180096a0 tx=0x7fda18006b40).stop 2024-02-20T13:47:11.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 msgr2=0x7fda281056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fda100089f0 tx=0x7fda1000b040).stop 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 shutdown_connections 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda14031090 0x7fda14033550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 --2- 172.21.15.192:0/1293407037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda280f5480 0x7fda281056f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 >> 172.21.15.192:0/1293407037 conn(0x7fda280f0be0 msgr2=0x7fda280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 shutdown_connections 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:11.982+0000 7fda2d0f1700 1 -- 172.21.15.192:0/1293407037 wait complete. 2024-02-20T13:47:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:12.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:11.981824+0000 mon.smithi158 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.192:0/1293407037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:13.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:11.382913+0000 mgr.smithi158.gmoobf (mgr.14156) 196 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:13.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:13.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:15.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:13.383161+0000 mgr.smithi158.gmoobf (mgr.14156) 197 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:15.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 -- 172.21.15.192:0/899921904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 msgr2=0x7fe6dc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/899921904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc0f58a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe6cc009fb0 tx=0x7fe6cc009b30).stop 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 -- 172.21.15.192:0/899921904 shutdown_connections 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/899921904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc0f58a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 -- 172.21.15.192:0/899921904 >> 172.21.15.192:0/899921904 conn(0x7fe6dc0f0be0 msgr2=0x7fe6dc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 -- 172.21.15.192:0/899921904 shutdown_connections 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.685+0000 7fe6e36ed700 1 -- 172.21.15.192:0/899921904 wait complete. 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e36ed700 1 Processor -- start 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e36ed700 1 -- start start 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e36ed700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:15.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e36ed700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc106cd0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e1489700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e1489700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52892/0 (socket says 172.21.15.192:52892) 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e1489700 1 -- 172.21.15.192:0/1527026482 learned_addr learned my addr 172.21.15.192:0/1527026482 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e1489700 1 -- 172.21.15.192:0/1527026482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc005040 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e1489700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe6cc0089f0 tx=0x7fe6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc009320 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.686+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc106ed0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.687+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe6cc013ec0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.687+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc014d30 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.687+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc107360 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.687+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe6cc0095f0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.688+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6dc045bb0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.688+0000 7fe6d1ffb700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 0x7fe6c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:15.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.688+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe6cc042490 con 0x7fe6dc0f5480 2024-02-20T13:47:15.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.688+0000 7fe6e0c88700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 0x7fe6c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:15.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.688+0000 7fe6e0c88700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 0x7fe6c8033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe6d80096a0 tx=0x7fe6d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:15.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.693+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe6cc01d7f0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.900+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6dc1081b0 con 0x7fe6dc0f5480 2024-02-20T13:47:15.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.900+0000 7fe6d1ffb700 1 -- 172.21.15.192:0/1527026482 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe6cc00fe40 con 0x7fe6dc0f5480 2024-02-20T13:47:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:15.888 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:15.889 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 msgr2=0x7fe6c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 0x7fe6c8033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe6d80096a0 tx=0x7fe6d8006b40).stop 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 msgr2=0x7fe6dc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe6cc0089f0 tx=0x7fe6cc00b040).stop 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 shutdown_connections 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe6c8031090 0x7fe6c8033550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 --2- 172.21.15.192:0/1527026482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc0f5480 0x7fe6dc1067e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 >> 172.21.15.192:0/1527026482 conn(0x7fe6dc0f0be0 msgr2=0x7fe6dc05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 shutdown_connections 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:15.902+0000 7fe6e36ed700 1 -- 172.21.15.192:0/1527026482 wait complete. 2024-02-20T13:47:15.890 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:16.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:15.901839+0000 mon.smithi158 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.192:0/1527026482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:17.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:15.383483+0000 mgr.smithi158.gmoobf (mgr.14156) 198 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:17.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:17.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:19.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:17.383842+0000 mgr.smithi158.gmoobf (mgr.14156) 199 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 -- 172.21.15.192:0/3160216466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 msgr2=0x7f24080f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 --2- 172.21.15.192:0/3160216466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24080f69b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f23fc004760 tx=0x7f23fc009ad0).stop 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 -- 172.21.15.192:0/3160216466 shutdown_connections 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 --2- 172.21.15.192:0/3160216466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24080f69b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 -- 172.21.15.192:0/3160216466 >> 172.21.15.192:0/3160216466 conn(0x7f24080f1cd0 msgr2=0x7f24080f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 -- 172.21.15.192:0/3160216466 shutdown_connections 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 -- 172.21.15.192:0/3160216466 wait complete. 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.548+0000 7f240e642700 1 Processor -- start 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f240e642700 1 -- start start 2024-02-20T13:47:19.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f240e642700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f240e642700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2408106cb0 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f2407fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f2407fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52898/0 (socket says 172.21.15.192:52898) 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f2407fff700 1 -- 172.21.15.192:0/3048363255 learned_addr learned my addr 172.21.15.192:0/3048363255 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f2407fff700 1 -- 172.21.15.192:0/3048363255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc009d70 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.549+0000 7f2407fff700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f23fc004170 tx=0x7f23fc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23fc00e040 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2408106eb0 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f23fc008ef0 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23fc014c90 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2408107340 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.550+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f23fc014420 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.551+0000 7f2404ff9700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 0x7f23f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.551+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2408045bb0 con 0x7f24080f6590 2024-02-20T13:47:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.551+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f23fc0422e0 con 0x7f24080f6590 2024-02-20T13:47:19.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.551+0000 7f24077fe700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 0x7f23f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:19.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.551+0000 7f24077fe700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 0x7f23f0033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f23f80096a0 tx=0x7f23f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:19.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.556+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f23fc026300 con 0x7f24080f6590 2024-02-20T13:47:19.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.762+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24080fc080 con 0x7f24080f6590 2024-02-20T13:47:19.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.763+0000 7f2404ff9700 1 -- 172.21.15.192:0/3048363255 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f23fc012b20 con 0x7f24080f6590 2024-02-20T13:47:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:19.750 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 msgr2=0x7f23f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 0x7f23f0033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f23f80096a0 tx=0x7f23f8006b40).stop 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 msgr2=0x7f24081067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f23fc004170 tx=0x7f23fc009ad0).stop 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 shutdown_connections 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f23f0031090 0x7f23f0033550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 --2- 172.21.15.192:0/3048363255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24080f6590 0x7f24081067c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 >> 172.21.15.192:0/3048363255 conn(0x7f24080f1cd0 msgr2=0x7f24080f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 shutdown_connections 2024-02-20T13:47:19.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:19.764+0000 7f240e642700 1 -- 172.21.15.192:0/3048363255 wait complete. 2024-02-20T13:47:19.753 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:20.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:19.764076+0000 mon.smithi158 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.192:0/3048363255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:21.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:21.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:21.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:19.384215+0000 mgr.smithi158.gmoobf (mgr.14156) 200 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:23.260 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 -- 172.21.15.192:0/2534199215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 msgr2=0x7f1ffc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 --2- 172.21.15.192:0/2534199215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc0f69b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1fec009fb0 tx=0x7f1fec009b30).stop 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 -- 172.21.15.192:0/2534199215 shutdown_connections 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 --2- 172.21.15.192:0/2534199215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc0f69b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 -- 172.21.15.192:0/2534199215 >> 172.21.15.192:0/2534199215 conn(0x7f1ffc0f1d10 msgr2=0x7f1ffc0f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 -- 172.21.15.192:0/2534199215 shutdown_connections 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.270+0000 7f20041a0700 1 -- 172.21.15.192:0/2534199215 wait complete. 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f20041a0700 1 Processor -- start 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f20041a0700 1 -- start start 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f20041a0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f20041a0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc106cd0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f2001f3c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f2001f3c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37038/0 (socket says 172.21.15.192:37038) 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f2001f3c700 1 -- 172.21.15.192:0/3914526283 learned_addr learned my addr 172.21.15.192:0/3914526283 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f2001f3c700 1 -- 172.21.15.192:0/3914526283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec005040 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f2001f3c700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1fec0089f0 tx=0x7f1fec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.271+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fec009320 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc106ed0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1fec013ec0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fec014d30 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc107360 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1fec0095f0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc045bb0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.272+0000 7f1ff27fc700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 0x7f1fe8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.273+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1fec042450 con 0x7f1ffc0f6590 2024-02-20T13:47:23.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.273+0000 7f200173b700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 0x7f1fe8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:23.263 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.273+0000 7f200173b700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 0x7f1fe8033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1ff80096a0 tx=0x7f1ff8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:23.265 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.278+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1fec01d7f0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.484+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ffc108210 con 0x7f1ffc0f6590 2024-02-20T13:47:23.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.484+0000 7f1ff27fc700 1 -- 172.21.15.192:0/3914526283 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1fec01d7f0 con 0x7f1ffc0f6590 2024-02-20T13:47:23.472 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:23.472 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:23.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 msgr2=0x7f1fe8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:23.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 0x7f1fe8033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1ff80096a0 tx=0x7f1ff8006b40).stop 2024-02-20T13:47:23.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 msgr2=0x7f1ffc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1fec0089f0 tx=0x7f1fec00b040).stop 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 shutdown_connections 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1fe8031090 0x7f1fe8033550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 --2- 172.21.15.192:0/3914526283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ffc0f6590 0x7f1ffc1067e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 >> 172.21.15.192:0/3914526283 conn(0x7f1ffc0f1d10 msgr2=0x7f1ffc0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 shutdown_connections 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:23.486+0000 7f20041a0700 1 -- 172.21.15.192:0/3914526283 wait complete. 2024-02-20T13:47:23.474 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:21.384615+0000 mgr.smithi158.gmoobf (mgr.14156) 201 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:24.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:23.485907+0000 mon.smithi158 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.192:0/3914526283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:25.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:25.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:25.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:23.385005+0000 mgr.smithi158.gmoobf (mgr.14156) 202 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:26.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 -- 172.21.15.192:0/3321030357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 msgr2=0x7f2ed80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/3321030357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed80f58a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f2ed4009fb0 tx=0x7f2ed4009b30).stop 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 -- 172.21.15.192:0/3321030357 shutdown_connections 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/3321030357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed80f58a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 -- 172.21.15.192:0/3321030357 >> 172.21.15.192:0/3321030357 conn(0x7f2ed80f0be0 msgr2=0x7f2ed80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 -- 172.21.15.192:0/3321030357 shutdown_connections 2024-02-20T13:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 -- 172.21.15.192:0/3321030357 wait complete. 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.965+0000 7f2ee00b4700 1 Processor -- start 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ee00b4700 1 -- start start 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ee00b4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ee00b4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed8060300 con 0x7f2ed80f5480 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2edde50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2edde50700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37044/0 (socket says 172.21.15.192:37044) 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2edde50700 1 -- 172.21.15.192:0/4076031208 learned_addr learned my addr 172.21.15.192:0/4076031208 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:26.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2edde50700 1 -- 172.21.15.192:0/4076031208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed4005040 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2edde50700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2ed40089f0 tx=0x7f2ed400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed4009320 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed8060500 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2ed4013ec0 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.966+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed4014d30 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.967+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed8109bc0 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.967+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2ed4014420 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.967+0000 7f2ece7fc700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 0x7f2ec4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.967+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2ed4042480 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.967+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed80591e0 con 0x7f2ed80f5480 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.968+0000 7f2edd64f700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 0x7f2ec4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:26.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.968+0000 7f2edd64f700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 0x7f2ec4033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2ec80096a0 tx=0x7f2ec8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:26.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:26.973+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2ed4014750 con 0x7f2ed80f5480 2024-02-20T13:47:27.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.179+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ed810a1d0 con 0x7f2ed80f5480 2024-02-20T13:47:27.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.180+0000 7f2ece7fc700 1 -- 172.21.15.192:0/4076031208 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2ed402b020 con 0x7f2ed80f5480 2024-02-20T13:47:27.168 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:27.168 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 msgr2=0x7f2ec4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 0x7f2ec4033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2ec80096a0 tx=0x7f2ec8006b40).stop 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 msgr2=0x7f2ed8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2ed40089f0 tx=0x7f2ed400b040).stop 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 shutdown_connections 2024-02-20T13:47:27.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ec4031090 0x7f2ec4033550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:27.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 --2- 172.21.15.192:0/4076031208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed80f5480 0x7f2ed8109510 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:27.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 >> 172.21.15.192:0/4076031208 conn(0x7f2ed80f0be0 msgr2=0x7f2ed80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:27.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 shutdown_connections 2024-02-20T13:47:27.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:27.182+0000 7f2ee00b4700 1 -- 172.21.15.192:0/4076031208 wait complete. 2024-02-20T13:47:27.170 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:27.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:25.385369+0000 mgr.smithi158.gmoobf (mgr.14156) 203 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:27.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:27.181580+0000 mon.smithi158 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.192:0/4076031208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:28.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:28.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:29.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:27.385661+0000 mgr.smithi158.gmoobf (mgr.14156) 204 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:30.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 -- 172.21.15.192:0/754695319 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 msgr2=0x7fa4400f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 --2- 172.21.15.192:0/754695319 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4400f69b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fa434009fb0 tx=0x7fa434009b30).stop 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 -- 172.21.15.192:0/754695319 shutdown_connections 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 --2- 172.21.15.192:0/754695319 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4400f69b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 -- 172.21.15.192:0/754695319 >> 172.21.15.192:0/754695319 conn(0x7fa4400f1d10 msgr2=0x7fa4400f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 -- 172.21.15.192:0/754695319 shutdown_connections 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.646+0000 7fa444894700 1 -- 172.21.15.192:0/754695319 wait complete. 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa444894700 1 Processor -- start 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa444894700 1 -- start start 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa444894700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa444894700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa440106cd0 con 0x7fa4400f6590 2024-02-20T13:47:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa43e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:30.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa43e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:37062/0 (socket says 172.21.15.192:37062) 2024-02-20T13:47:30.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.647+0000 7fa43e59c700 1 -- 172.21.15.192:0/4002594722 learned_addr learned my addr 172.21.15.192:0/4002594722 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa43e59c700 1 -- 172.21.15.192:0/4002594722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434005040 con 0x7fa4400f6590 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa43e59c700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa4340089f0 tx=0x7fa43400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa434009320 con 0x7fa4400f6590 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa440106ed0 con 0x7fa4400f6590 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440107360 con 0x7fa4400f6590 2024-02-20T13:47:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa434013ec0 con 0x7fa4400f6590 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.648+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa434014d00 con 0x7fa4400f6590 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.649+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa434014420 con 0x7fa4400f6590 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.649+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa424004fa0 con 0x7fa4400f6590 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.649+0000 7fa42effd700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 0x7fa4280335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.649+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa434042350 con 0x7fa4400f6590 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.649+0000 7fa43dd9b700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 0x7fa4280335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.650+0000 7fa43dd9b700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 0x7fa4280335a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fa4300096a0 tx=0x7fa430006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:30.640 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.654+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa434026c40 con 0x7fa4400f6590 2024-02-20T13:47:30.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.860+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa424005670 con 0x7fa4400f6590 2024-02-20T13:47:30.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.860+0000 7fa42effd700 1 -- 172.21.15.192:0/4002594722 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa434012b20 con 0x7fa4400f6590 2024-02-20T13:47:30.847 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:30.847 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 msgr2=0x7fa4280335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 0x7fa4280335a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fa4300096a0 tx=0x7fa430006b40).stop 2024-02-20T13:47:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 msgr2=0x7fa4401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa4340089f0 tx=0x7fa43400b040).stop 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 shutdown_connections 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa4280310e0 0x7fa4280335a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 --2- 172.21.15.192:0/4002594722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4400f6590 0x7fa4401067e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 >> 172.21.15.192:0/4002594722 conn(0x7fa4400f1d10 msgr2=0x7fa4400f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 shutdown_connections 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:30.862+0000 7fa444894700 1 -- 172.21.15.192:0/4002594722 wait complete. 2024-02-20T13:47:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:31.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:29.386000+0000 mgr.smithi158.gmoobf (mgr.14156) 205 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:31.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:30.860913+0000 mon.smithi158 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.192:0/4002594722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:32.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:32.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:33.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:31.386334+0000 mgr.smithi158.gmoobf (mgr.14156) 206 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:34.571 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.584+0000 7f14ecfeb700 1 -- 172.21.15.192:0/1497219426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 msgr2=0x7f14e80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:34.571 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.584+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/1497219426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e80f58a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f14dc009fb0 tx=0x7f14dc009b30).stop 2024-02-20T13:47:34.571 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 -- 172.21.15.192:0/1497219426 shutdown_connections 2024-02-20T13:47:34.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/1497219426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e80f58a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:34.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 -- 172.21.15.192:0/1497219426 >> 172.21.15.192:0/1497219426 conn(0x7f14e80f0be0 msgr2=0x7f14e80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:34.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 -- 172.21.15.192:0/1497219426 shutdown_connections 2024-02-20T13:47:34.572 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 -- 172.21.15.192:0/1497219426 wait complete. 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 Processor -- start 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.585+0000 7f14ecfeb700 1 -- start start 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14ecfeb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14ecfeb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8105be0 con 0x7f14e80f5480 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14e659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14e659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52692/0 (socket says 172.21.15.192:52692) 2024-02-20T13:47:34.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14e659c700 1 -- 172.21.15.192:0/4201602271 learned_addr learned my addr 172.21.15.192:0/4201602271 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14e659c700 1 -- 172.21.15.192:0/4201602271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc005040 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14e659c700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f14dc0089f0 tx=0x7f14dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14dc009320 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e8105de0 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f14dc013ec0 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14dc014d30 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.586+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e8106270 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.587+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f14dc014420 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.587+0000 7f14d6ffd700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 0x7f14d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.587+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f14dc0423c0 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.587+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8045bb0 con 0x7f14e80f5480 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.588+0000 7f14e5d9b700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 0x7f14d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.588+0000 7f14e5d9b700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 0x7f14d0033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f14d80096a0 tx=0x7f14d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:34.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.592+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f14dc014750 con 0x7f14e80f5480 2024-02-20T13:47:34.785 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.798+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e80faf70 con 0x7f14e80f5480 2024-02-20T13:47:34.785 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.799+0000 7f14d6ffd700 1 -- 172.21.15.192:0/4201602271 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14dc02b020 con 0x7f14e80f5480 2024-02-20T13:47:34.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:34.786 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:34.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 msgr2=0x7f14d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:34.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 0x7f14d0033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f14d80096a0 tx=0x7f14d8006b40).stop 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 msgr2=0x7f14e81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f14dc0089f0 tx=0x7f14dc00b040).stop 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 shutdown_connections 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f14d0031090 0x7f14d0033550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 --2- 172.21.15.192:0/4201602271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14e80f5480 0x7f14e81056f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 >> 172.21.15.192:0/4201602271 conn(0x7f14e80f0be0 msgr2=0x7f14e80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 shutdown_connections 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:34.801+0000 7f14ecfeb700 1 -- 172.21.15.192:0/4201602271 wait complete. 2024-02-20T13:47:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:35.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:33.386721+0000 mgr.smithi158.gmoobf (mgr.14156) 207 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:35.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:34.799743+0000 mon.smithi158 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.192:0/4201602271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:36.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:36.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:37.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:35.387158+0000 mgr.smithi158.gmoobf (mgr.14156) 208 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:38.367 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 -- 172.21.15.192:0/1910199296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 msgr2=0x7fa2900f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 --2- 172.21.15.192:0/1910199296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2900f69a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fa280009fb0 tx=0x7fa280009b30).stop 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 -- 172.21.15.192:0/1910199296 shutdown_connections 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 --2- 172.21.15.192:0/1910199296 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2900f69a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 -- 172.21.15.192:0/1910199296 >> 172.21.15.192:0/1910199296 conn(0x7fa2900f1ce0 msgr2=0x7fa2900f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 -- 172.21.15.192:0/1910199296 shutdown_connections 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.377+0000 7fa297a5d700 1 -- 172.21.15.192:0/1910199296 wait complete. 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa297a5d700 1 Processor -- start 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa297a5d700 1 -- start start 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa297a5d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa297a5d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa290106ca0 con 0x7fa2900f6580 2024-02-20T13:47:38.368 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa2957f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa2957f9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52718/0 (socket says 172.21.15.192:52718) 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa2957f9700 1 -- 172.21.15.192:0/475814503 learned_addr learned my addr 172.21.15.192:0/475814503 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.378+0000 7fa2957f9700 1 -- 172.21.15.192:0/475814503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280005040 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa2957f9700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fa2800089f0 tx=0x7fa28000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa280009320 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa290106ea0 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa280013ec0 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa280014d30 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.379+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa290107330 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.380+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa280014420 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.380+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa290045bb0 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.380+0000 7fa285ffb700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 0x7fa27c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.380+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa280042390 con 0x7fa2900f6580 2024-02-20T13:47:38.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.380+0000 7fa294ff8700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 0x7fa27c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:38.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.381+0000 7fa294ff8700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 0x7fa27c033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa28c0096a0 tx=0x7fa28c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:38.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.385+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2800268c0 con 0x7fa2900f6580 2024-02-20T13:47:38.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.593+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2901081e0 con 0x7fa2900f6580 2024-02-20T13:47:38.579 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.593+0000 7fa285ffb700 1 -- 172.21.15.192:0/475814503 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa28001d7f0 con 0x7fa2900f6580 2024-02-20T13:47:38.580 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:38.580 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 msgr2=0x7fa27c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 0x7fa27c033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa28c0096a0 tx=0x7fa28c006b40).stop 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 msgr2=0x7fa2901067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fa2800089f0 tx=0x7fa28000b040).stop 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 shutdown_connections 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa27c031090 0x7fa27c033550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:38.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 --2- 172.21.15.192:0/475814503 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2900f6580 0x7fa2901067b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 >> 172.21.15.192:0/475814503 conn(0x7fa2900f1ce0 msgr2=0x7fa2900f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 shutdown_connections 2024-02-20T13:47:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:38.595+0000 7fa297a5d700 1 -- 172.21.15.192:0/475814503 wait complete. 2024-02-20T13:47:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:39.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:37.387442+0000 mgr.smithi158.gmoobf (mgr.14156) 209 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:39.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:38.593901+0000 mon.smithi158 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.192:0/475814503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:40.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:40.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:41.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:39.387759+0000 mgr.smithi158.gmoobf (mgr.14156) 210 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 -- 172.21.15.192:0/243591023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 msgr2=0x7f69380f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 --2- 172.21.15.192:0/243591023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f69380f58a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f6928009fb0 tx=0x7f6928009b30).stop 2024-02-20T13:47:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 -- 172.21.15.192:0/243591023 shutdown_connections 2024-02-20T13:47:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 --2- 172.21.15.192:0/243591023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f69380f58a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 -- 172.21.15.192:0/243591023 >> 172.21.15.192:0/243591023 conn(0x7f69380f0be0 msgr2=0x7f69380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:42.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.155+0000 7f693e2c7700 1 -- 172.21.15.192:0/243591023 shutdown_connections 2024-02-20T13:47:42.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.156+0000 7f693e2c7700 1 -- 172.21.15.192:0/243591023 wait complete. 2024-02-20T13:47:42.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.156+0000 7f693e2c7700 1 Processor -- start 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.156+0000 7f693e2c7700 1 -- start start 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.156+0000 7f693e2c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.156+0000 7f693e2c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938060300 con 0x7f69380f5480 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6937fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6937fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:52728/0 (socket says 172.21.15.192:52728) 2024-02-20T13:47:42.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6937fff700 1 -- 172.21.15.192:0/1261021808 learned_addr learned my addr 172.21.15.192:0/1261021808 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6937fff700 1 -- 172.21.15.192:0/1261021808 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928005040 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6937fff700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6928000c00 tx=0x7f692800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69280095f0 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.157+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6938060500 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.158+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6938109bc0 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.158+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6928007e50 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.158+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6928014da0 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.158+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6928025070 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.158+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69380591e0 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.159+0000 7f6934ff9700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 0x7f6920033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.159+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6928041de0 con 0x7f69380f5480 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.159+0000 7f69377fe700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 0x7f6920033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.159+0000 7f69377fe700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 0x7f6920033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f692c0096a0 tx=0x7f692c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:42.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.163+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f69280254e0 con 0x7f69380f5480 2024-02-20T13:47:42.358 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.371+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6938045bb0 con 0x7f69380f5480 2024-02-20T13:47:42.359 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.371+0000 7f6934ff9700 1 -- 172.21.15.192:0/1261021808 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6938045bb0 con 0x7f69380f5480 2024-02-20T13:47:42.359 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:42.359 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 msgr2=0x7f6920033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 0x7f6920033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f692c0096a0 tx=0x7f692c006b40).stop 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 msgr2=0x7f6938109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6928000c00 tx=0x7f692800b040).stop 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 shutdown_connections 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6920031090 0x7f6920033550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 --2- 172.21.15.192:0/1261021808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69380f5480 0x7f6938109510 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 >> 172.21.15.192:0/1261021808 conn(0x7f69380f0be0 msgr2=0x7f69380f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:42.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 shutdown_connections 2024-02-20T13:47:42.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:42.373+0000 7f693e2c7700 1 -- 172.21.15.192:0/1261021808 wait complete. 2024-02-20T13:47:42.362 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:43.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:41.388114+0000 mgr.smithi158.gmoobf (mgr.14156) 211 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:43.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:42.373004+0000 mon.smithi158 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.192:0/1261021808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:44.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:44.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:45.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:43.388493+0000 mgr.smithi158.gmoobf (mgr.14156) 212 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- 172.21.15.192:0/4266613538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 msgr2=0x7fc4e00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/4266613538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e00f58a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc4d4009fb0 tx=0x7fc4d4009b30).stop 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- 172.21.15.192:0/4266613538 shutdown_connections 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/4266613538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e00f58a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- 172.21.15.192:0/4266613538 >> 172.21.15.192:0/4266613538 conn(0x7fc4e00f0be0 msgr2=0x7fc4e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- 172.21.15.192:0/4266613538 shutdown_connections 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- 172.21.15.192:0/4266613538 wait complete. 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 Processor -- start 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.876+0000 7fc4e6d86700 1 -- start start 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e6d86700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e6d86700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0105be0 con 0x7fc4e00f5480 2024-02-20T13:47:45.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e4b22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e4b22700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40818/0 (socket says 172.21.15.192:40818) 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e4b22700 1 -- 172.21.15.192:0/3711068235 learned_addr learned my addr 172.21.15.192:0/3711068235 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e4b22700 1 -- 172.21.15.192:0/3711068235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4005040 con 0x7fc4e00f5480 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e4b22700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fc4d4000c00 tx=0x7fc4d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d40095f0 con 0x7fc4e00f5480 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e0105de0 con 0x7fc4e00f5480 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc4d4007e50 con 0x7fc4e00f5480 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d4014e70 con 0x7fc4e00f5480 2024-02-20T13:47:45.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.877+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e0106270 con 0x7fc4e00f5480 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.878+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc4d4025070 con 0x7fc4e00f5480 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.878+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0045bb0 con 0x7fc4e00f5480 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.878+0000 7fc4dd7fa700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 0x7fc4d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.878+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc4d4041d50 con 0x7fc4e00f5480 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.879+0000 7fc4dffff700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 0x7fc4d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:45.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.879+0000 7fc4dffff700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 0x7fc4d0033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc4cc0096a0 tx=0x7fc4cc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:45.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:45.883+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc4d400f4e0 con 0x7fc4e00f5480 2024-02-20T13:47:46.077 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.089+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4e00faf70 con 0x7fc4e00f5480 2024-02-20T13:47:46.077 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.090+0000 7fc4dd7fa700 1 -- 172.21.15.192:0/3711068235 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc4d400f300 con 0x7fc4e00f5480 2024-02-20T13:47:46.077 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:46.077 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 msgr2=0x7fc4d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 0x7fc4d0033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc4cc0096a0 tx=0x7fc4cc006b40).stop 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 msgr2=0x7fc4e01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fc4d4000c00 tx=0x7fc4d400b040).stop 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 shutdown_connections 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc4d0031090 0x7fc4d0033550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 --2- 172.21.15.192:0/3711068235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4e00f5480 0x7fc4e01056f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 >> 172.21.15.192:0/3711068235 conn(0x7fc4e00f0be0 msgr2=0x7fc4e00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:46.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 shutdown_connections 2024-02-20T13:47:46.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:46.092+0000 7fc4e6d86700 1 -- 172.21.15.192:0/3711068235 wait complete. 2024-02-20T13:47:46.080 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:46.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:46.091578+0000 mon.smithi158 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.192:0/3711068235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:47.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:45.388875+0000 mgr.smithi158.gmoobf (mgr.14156) 213 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:47.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:47.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:49.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:47.389139+0000 mgr.smithi158.gmoobf (mgr.14156) 214 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 -- 172.21.15.192:0/3221501818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 msgr2=0x7fe4f40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 --2- 172.21.15.192:0/3221501818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f40f69b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe4e4009fb0 tx=0x7fe4e4009b30).stop 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 -- 172.21.15.192:0/3221501818 shutdown_connections 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 --2- 172.21.15.192:0/3221501818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f40f69b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 -- 172.21.15.192:0/3221501818 >> 172.21.15.192:0/3221501818 conn(0x7fe4f40f1d10 msgr2=0x7fe4f40f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 -- 172.21.15.192:0/3221501818 shutdown_connections 2024-02-20T13:47:49.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.765+0000 7fe4fb654700 1 -- 172.21.15.192:0/3221501818 wait complete. 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4fb654700 1 Processor -- start 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4fb654700 1 -- start start 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4fb654700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4fb654700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4106cc0 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4f93f0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4f93f0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40840/0 (socket says 172.21.15.192:40840) 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.766+0000 7fe4f93f0700 1 -- 172.21.15.192:0/4085871535 learned_addr learned my addr 172.21.15.192:0/4085871535 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4f93f0700 1 -- 172.21.15.192:0/4085871535 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4005040 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4f93f0700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe4e40089f0 tx=0x7fe4e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e4009320 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f4106ec0 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe4e4013ec0 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e4014d30 con 0x7fe4f40f6590 2024-02-20T13:47:49.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.767+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f4107350 con 0x7fe4f40f6590 2024-02-20T13:47:49.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.768+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe4e40095f0 con 0x7fe4f40f6590 2024-02-20T13:47:49.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.768+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f4045bb0 con 0x7fe4f40f6590 2024-02-20T13:47:49.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.768+0000 7fe4e9ffb700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 0x7fe4e00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:49.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.768+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe4e4042450 con 0x7fe4f40f6590 2024-02-20T13:47:49.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.769+0000 7fe4f8bef700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 0x7fe4e00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:49.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.769+0000 7fe4f8bef700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 0x7fe4e00335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:49.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.773+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe4e401d7f0 con 0x7fe4f40f6590 2024-02-20T13:47:49.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.978+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4f40fc080 con 0x7fe4f40f6590 2024-02-20T13:47:49.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.979+0000 7fe4e9ffb700 1 -- 172.21.15.192:0/4085871535 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4e401d7f0 con 0x7fe4f40f6590 2024-02-20T13:47:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:49.966 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:49.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 msgr2=0x7fe4e00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 0x7fe4e00335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).stop 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 msgr2=0x7fe4f41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe4e40089f0 tx=0x7fe4e400b040).stop 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 shutdown_connections 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe4e00310e0 0x7fe4e00335a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 --2- 172.21.15.192:0/4085871535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4f40f6590 0x7fe4f41067d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 >> 172.21.15.192:0/4085871535 conn(0x7fe4f40f1d10 msgr2=0x7fe4f40f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 shutdown_connections 2024-02-20T13:47:49.968 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:49.980+0000 7fe4fb654700 1 -- 172.21.15.192:0/4085871535 wait complete. 2024-02-20T13:47:49.969 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:50.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:49.979992+0000 mon.smithi158 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.192:0/4085871535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:51.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:49.389471+0000 mgr.smithi158.gmoobf (mgr.14156) 215 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:51.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:51.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:53.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:51.389891+0000 mgr.smithi158.gmoobf (mgr.14156) 216 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.735+0000 7f602dd24700 1 -- 172.21.15.192:0/867296181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 msgr2=0x7f60280f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.735+0000 7f602dd24700 1 --2- 172.21.15.192:0/867296181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60280f4c60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6018009fb0 tx=0x7f6018009b30).stop 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.735+0000 7f602dd24700 1 -- 172.21.15.192:0/867296181 shutdown_connections 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.735+0000 7f602dd24700 1 --2- 172.21.15.192:0/867296181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60280f4c60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.735+0000 7f602dd24700 1 -- 172.21.15.192:0/867296181 >> 172.21.15.192:0/867296181 conn(0x7f60280f0360 msgr2=0x7f60280f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 -- 172.21.15.192:0/867296181 shutdown_connections 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 -- 172.21.15.192:0/867296181 wait complete. 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 Processor -- start 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 -- start start 2024-02-20T13:47:53.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.736+0000 7f602dd24700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6028106cb0 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f60277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f60277fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:59990/0 (socket says 172.21.15.192:59990) 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f60277fe700 1 -- 172.21.15.192:0/4281630569 learned_addr learned my addr 172.21.15.192:0/4281630569 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f60277fe700 1 -- 172.21.15.192:0/4281630569 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018005040 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f60277fe700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f60180089f0 tx=0x7f601800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6018009320 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6028106eb0 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6018013ec0 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6018014d30 con 0x7f60280f4840 2024-02-20T13:47:53.726 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.737+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6028107340 con 0x7f60280f4840 2024-02-20T13:47:53.727 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.738+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f60180263a0 con 0x7f60280f4840 2024-02-20T13:47:53.727 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.738+0000 7f600ffff700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 0x7f6010033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:53.728 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.738+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6018042440 con 0x7f60280f4840 2024-02-20T13:47:53.728 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.739+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60280591e0 con 0x7f60280f4840 2024-02-20T13:47:53.728 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.739+0000 7f6026ffd700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 0x7f6010033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:53.728 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.739+0000 7f6026ffd700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 0x7f6010033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f601c0096a0 tx=0x7f601c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:53.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.743+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f60180266d0 con 0x7f60280f4840 2024-02-20T13:47:53.937 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.950+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6028045bb0 con 0x7f60280f4840 2024-02-20T13:47:53.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.950+0000 7f600ffff700 1 -- 172.21.15.192:0/4281630569 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f601802b020 con 0x7f60280f4840 2024-02-20T13:47:53.938 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:53.938 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:53.939 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 msgr2=0x7f6010033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:53.939 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 0x7f6010033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f601c0096a0 tx=0x7f601c006b40).stop 2024-02-20T13:47:53.939 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 msgr2=0x7f60281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:53.939 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f60180089f0 tx=0x7f601800b040).stop 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 shutdown_connections 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6010031090 0x7f6010033550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 --2- 172.21.15.192:0/4281630569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60280f4840 0x7f60281067c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 >> 172.21.15.192:0/4281630569 conn(0x7f60280f0360 msgr2=0x7f60280f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 shutdown_connections 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:53.952+0000 7f602dd24700 1 -- 172.21.15.192:0/4281630569 wait complete. 2024-02-20T13:47:53.940 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:54.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:54 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:53.951969+0000 mon.smithi158 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.192:0/4281630569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:55.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:53.390272+0000 mgr.smithi158.gmoobf (mgr.14156) 217 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:55.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:55.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:47:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:55.465073+0000 mon.smithi158 (mon.0) 236 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:47:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:55.465281+0000 mon.smithi158 (mon.0) 237 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:47:57.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:55.390643+0000 mgr.smithi158.gmoobf (mgr.14156) 218 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- 172.21.15.192:0/1983182119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 msgr2=0x7fa9900f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 --2- 172.21.15.192:0/1983182119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f91c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fa98c009fb0 tx=0x7fa98c009b30).stop 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- 172.21.15.192:0/1983182119 shutdown_connections 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 --2- 172.21.15.192:0/1983182119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f91c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- 172.21.15.192:0/1983182119 >> 172.21.15.192:0/1983182119 conn(0x7fa9900f0be0 msgr2=0x7fa9900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- 172.21.15.192:0/1983182119 shutdown_connections 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- 172.21.15.192:0/1983182119 wait complete. 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 Processor -- start 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.661+0000 7fa997311700 1 -- start start 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa997311700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa997311700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900f50c0 con 0x7fa990060300 2024-02-20T13:47:57.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa9950ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:57.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa9950ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:60014/0 (socket says 172.21.15.192:60014) 2024-02-20T13:47:57.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa9950ad700 1 -- 172.21.15.192:0/1522966365 learned_addr learned my addr 172.21.15.192:0/1522966365 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:47:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa9950ad700 1 -- 172.21.15.192:0/1522966365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c005040 con 0x7fa990060300 2024-02-20T13:47:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa9950ad700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa98c009f80 tx=0x7fa98c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa98c009320 con 0x7fa990060300 2024-02-20T13:47:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9900f5260 con 0x7fa990060300 2024-02-20T13:47:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa98c013680 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.662+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa98c01eda0 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.663+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9900f56f0 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa98c01e420 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9900591e0 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa985ffb700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 0x7fa97c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa98c042360 con 0x7fa990060300 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa9948ac700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 0x7fa97c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:47:57.652 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.664+0000 7fa9948ac700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 0x7fa97c033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa9800096a0 tx=0x7fa980006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:47:57.655 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.669+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa98c01d380 con 0x7fa990060300 2024-02-20T13:47:57.861 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.874+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa990045bb0 con 0x7fa990060300 2024-02-20T13:47:57.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.875+0000 7fa985ffb700 1 -- 172.21.15.192:0/1522966365 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa98c01cb10 con 0x7fa990060300 2024-02-20T13:47:57.862 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:47:57.863 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 msgr2=0x7fa97c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 0x7fa97c033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa9800096a0 tx=0x7fa980006b40).stop 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 msgr2=0x7fa9900f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa98c009f80 tx=0x7fa98c00b040).stop 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 shutdown_connections 2024-02-20T13:47:57.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa97c031090 0x7fa97c033550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:57.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 --2- 172.21.15.192:0/1522966365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa990060300 0x7fa9900f6c40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:47:57.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 >> 172.21.15.192:0/1522966365 conn(0x7fa9900f0be0 msgr2=0x7fa9900f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:47:57.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 shutdown_connections 2024-02-20T13:47:57.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:47:57.877+0000 7fa997311700 1 -- 172.21.15.192:0/1522966365 wait complete. 2024-02-20T13:47:57.865 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:47:58.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:47:57.876612+0000 mon.smithi158 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.192:0/1522966365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:47:59.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:47:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:57.391030+0000 mgr.smithi158.gmoobf (mgr.14156) 219 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:47:59.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:47:59.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:01.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:47:59.391455+0000 mgr.smithi158.gmoobf (mgr.14156) 220 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:01.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/595909322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 msgr2=0x7fbd280fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:01.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/595909322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd280fb5c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbd24009fb0 tx=0x7fbd24009b30).stop 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/595909322 shutdown_connections 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/595909322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd280fb5c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/595909322 >> 172.21.15.192:0/595909322 conn(0x7fbd280f0be0 msgr2=0x7fbd280f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/595909322 shutdown_connections 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.567+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/595909322 wait complete. 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2f9ec700 1 Processor -- start 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2f9ec700 1 -- start start 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2f9ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2f9ec700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28108e60 con 0x7fbd28060300 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2d788700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2d788700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:60028/0 (socket says 172.21.15.192:60028) 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2d788700 1 -- 172.21.15.192:0/3683367981 learned_addr learned my addr 172.21.15.192:0/3683367981 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:01.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.568+0000 7fbd2d788700 1 -- 172.21.15.192:0/3683367981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24005040 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd2d788700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbd24009f80 tx=0x7fbd2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd24009320 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd28109060 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd24013680 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd2401eda0 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.569+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd281094f0 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fbd2401e420 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd1dffb700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 0x7fbd14033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd28045bb0 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd24042300 con 0x7fbd28060300 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd2cf87700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 0x7fbd14033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.570+0000 7fbd2cf87700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 0x7fbd14033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbd180096a0 tx=0x7fbd18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:01.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.575+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd2401ccf0 con 0x7fbd28060300 2024-02-20T13:48:01.767 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.780+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd2810a3a0 con 0x7fbd28060300 2024-02-20T13:48:01.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.781+0000 7fbd1dffb700 1 -- 172.21.15.192:0/3683367981 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbd2401cb10 con 0x7fbd28060300 2024-02-20T13:48:01.768 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:01.768 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 msgr2=0x7fbd14033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 0x7fbd14033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbd180096a0 tx=0x7fbd18006b40).stop 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 msgr2=0x7fbd28108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbd24009f80 tx=0x7fbd2400b040).stop 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 shutdown_connections 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbd14031090 0x7fbd14033550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 --2- 172.21.15.192:0/3683367981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd28060300 0x7fbd28108970 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 >> 172.21.15.192:0/3683367981 conn(0x7fbd280f0be0 msgr2=0x7fbd280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:01.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 shutdown_connections 2024-02-20T13:48:01.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:01.783+0000 7fbd2f9ec700 1 -- 172.21.15.192:0/3683367981 wait complete. 2024-02-20T13:48:01.771 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:02.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:01.782471+0000 mon.smithi158 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.192:0/3683367981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:03.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:03.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:03.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:01.391847+0000 mgr.smithi158.gmoobf (mgr.14156) 221 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/4245398875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 msgr2=0x7f3fc40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/4245398875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc40f58a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f3fb4009fb0 tx=0x7f3fb4009b30).stop 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/4245398875 shutdown_connections 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/4245398875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc40f58a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/4245398875 >> 172.21.15.192:0/4245398875 conn(0x7f3fc40f0be0 msgr2=0x7f3fc40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/4245398875 shutdown_connections 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.374+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/4245398875 wait complete. 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc8ac5700 1 Processor -- start 2024-02-20T13:48:05.363 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc8ac5700 1 -- start start 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc8ac5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc8ac5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4106ce0 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58474/0 (socket says 172.21.15.192:58474) 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc259c700 1 -- 172.21.15.192:0/1575366633 learned_addr learned my addr 172.21.15.192:0/1575366633 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.375+0000 7f3fc259c700 1 -- 172.21.15.192:0/1575366633 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb4005040 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fc259c700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3fb40089f0 tx=0x7f3fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb4009320 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc4106ee0 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fb4013ec0 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb4014d30 con 0x7f3fc40f5480 2024-02-20T13:48:05.364 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.376+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc4107370 con 0x7f3fc40f5480 2024-02-20T13:48:05.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3fb40095f0 con 0x7f3fc40f5480 2024-02-20T13:48:05.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc4045bb0 con 0x7f3fc40f5480 2024-02-20T13:48:05.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fbaffd700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 0x7f3fb0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:05.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fb40423c0 con 0x7f3fc40f5480 2024-02-20T13:48:05.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fc1d9b700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 0x7f3fb0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:05.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.377+0000 7f3fc1d9b700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 0x7f3fb0033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f3fac0096a0 tx=0x7f3fac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:05.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.382+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fb40268c0 con 0x7f3fc40f5480 2024-02-20T13:48:05.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:03.392127+0000 mgr.smithi158.gmoobf (mgr.14156) 222 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:05.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.587+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fc40faf70 con 0x7f3fc40f5480 2024-02-20T13:48:05.574 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.587+0000 7f3fbaffd700 1 -- 172.21.15.192:0/1575366633 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3fb401d380 con 0x7f3fc40f5480 2024-02-20T13:48:05.574 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:05.575 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 msgr2=0x7f3fb0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 0x7f3fb0033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f3fac0096a0 tx=0x7f3fac006b40).stop 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 msgr2=0x7f3fc41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3fb40089f0 tx=0x7f3fb400b040).stop 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 shutdown_connections 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3fb0031090 0x7f3fb0033550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 --2- 172.21.15.192:0/1575366633 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fc40f5480 0x7f3fc41067f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 >> 172.21.15.192:0/1575366633 conn(0x7f3fc40f0be0 msgr2=0x7f3fc40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 shutdown_connections 2024-02-20T13:48:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:05.589+0000 7f3fc8ac5700 1 -- 172.21.15.192:0/1575366633 wait complete. 2024-02-20T13:48:05.577 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:06.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:06 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:05.588725+0000 mon.smithi158 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.192:0/1575366633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:07.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:07.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:07.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:05.392406+0000 mgr.smithi158.gmoobf (mgr.14156) 223 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 -- 172.21.15.192:0/2256807112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 msgr2=0x7f19480f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 --2- 172.21.15.192:0/2256807112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19480f58d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f1938009fb0 tx=0x7f1938009b30).stop 2024-02-20T13:48:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 -- 172.21.15.192:0/2256807112 shutdown_connections 2024-02-20T13:48:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 --2- 172.21.15.192:0/2256807112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19480f58d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 -- 172.21.15.192:0/2256807112 >> 172.21.15.192:0/2256807112 conn(0x7f19480f0c10 msgr2=0x7f19480f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 -- 172.21.15.192:0/2256807112 shutdown_connections 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.160+0000 7f194f62f700 1 -- 172.21.15.192:0/2256807112 wait complete. 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194f62f700 1 Processor -- start 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194f62f700 1 -- start start 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194f62f700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194f62f700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948106ca0 con 0x7f19480f54b0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194d3cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194d3cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58498/0 (socket says 172.21.15.192:58498) 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194d3cb700 1 -- 172.21.15.192:0/3566374648 learned_addr learned my addr 172.21.15.192:0/3566374648 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.161+0000 7f194d3cb700 1 -- 172.21.15.192:0/3566374648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938005040 con 0x7f19480f54b0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f194d3cb700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f19380089f0 tx=0x7f193800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1938009320 con 0x7f19480f54b0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1938013ec0 con 0x7f19480f54b0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1948106ea0 con 0x7f19480f54b0 2024-02-20T13:48:09.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1938014d30 con 0x7f19480f54b0 2024-02-20T13:48:09.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.162+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1948107330 con 0x7f19480f54b0 2024-02-20T13:48:09.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.163+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f19380095f0 con 0x7f19480f54b0 2024-02-20T13:48:09.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.163+0000 7f193dffb700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 0x7f1934033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:09.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.163+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1948045bb0 con 0x7f19480f54b0 2024-02-20T13:48:09.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.163+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1938042450 con 0x7f19480f54b0 2024-02-20T13:48:09.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.163+0000 7f194cbca700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 0x7f1934033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:09.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.164+0000 7f194cbca700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 0x7f1934033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f19440096a0 tx=0x7f1944006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:09.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.168+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f193801d7f0 con 0x7f19480f54b0 2024-02-20T13:48:09.359 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.372+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19480fb090 con 0x7f19480f54b0 2024-02-20T13:48:09.360 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.373+0000 7f193dffb700 1 -- 172.21.15.192:0/3566374648 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f193801d7f0 con 0x7f19480f54b0 2024-02-20T13:48:09.360 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:09.360 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:09.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 msgr2=0x7f1934033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 0x7f1934033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f19440096a0 tx=0x7f1944006b40).stop 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 msgr2=0x7f19481067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f19380089f0 tx=0x7f193800b040).stop 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 shutdown_connections 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.374+0000 7f194f62f700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1934031090 0x7f1934033550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.375+0000 7f194f62f700 1 --2- 172.21.15.192:0/3566374648 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19480f54b0 0x7f19481067b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.375+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 >> 172.21.15.192:0/3566374648 conn(0x7f19480f0c10 msgr2=0x7f194805ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.375+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 shutdown_connections 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:09.375+0000 7f194f62f700 1 -- 172.21.15.192:0/3566374648 wait complete. 2024-02-20T13:48:09.362 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:09.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:07.392735+0000 mgr.smithi158.gmoobf (mgr.14156) 224 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:10.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:09.374301+0000 mon.smithi158 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.192:0/3566374648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:11.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:11.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:11.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:09.393045+0000 mgr.smithi158.gmoobf (mgr.14156) 225 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:12.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:11.393323+0000 mgr.smithi158.gmoobf (mgr.14156) 226 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 -- 172.21.15.192:0/1543576351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 msgr2=0x7fe6540f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1543576351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6540f4c70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fe648009fb0 tx=0x7fe648009b30).stop 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 -- 172.21.15.192:0/1543576351 shutdown_connections 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1543576351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6540f4c70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 -- 172.21.15.192:0/1543576351 >> 172.21.15.192:0/1543576351 conn(0x7fe6540f0390 msgr2=0x7fe6540f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 -- 172.21.15.192:0/1543576351 shutdown_connections 2024-02-20T13:48:12.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.963+0000 7fe6598d5700 1 -- 172.21.15.192:0/1543576351 wait complete. 2024-02-20T13:48:12.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe6598d5700 1 Processor -- start 2024-02-20T13:48:12.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe6598d5700 1 -- start start 2024-02-20T13:48:12.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe6598d5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:12.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe6598d5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654106c90 con 0x7fe6540f4850 2024-02-20T13:48:12.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe652ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe652ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58634/0 (socket says 172.21.15.192:58634) 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.964+0000 7fe652ffd700 1 -- 172.21.15.192:0/1494452924 learned_addr learned my addr 172.21.15.192:0/1494452924 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe652ffd700 1 -- 172.21.15.192:0/1494452924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648005040 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe652ffd700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fe6480089f0 tx=0x7fe64800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe648009320 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe648013ec0 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe654106e90 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe648014d30 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.965+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe654107320 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.966+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe6480095f0 con 0x7fe6540f4850 2024-02-20T13:48:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.966+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654045bb0 con 0x7fe6540f4850 2024-02-20T13:48:12.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.966+0000 7fe63b7fe700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 0x7fe63c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:12.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.966+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe648042450 con 0x7fe6540f4850 2024-02-20T13:48:12.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.966+0000 7fe6527fc700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 0x7fe63c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:12.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.967+0000 7fe6527fc700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 0x7fe63c033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe6440096a0 tx=0x7fe644006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:12.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:12.971+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe64801d7f0 con 0x7fe6540f4850 2024-02-20T13:48:13.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.180+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6541081d0 con 0x7fe6540f4850 2024-02-20T13:48:13.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.180+0000 7fe63b7fe700 1 -- 172.21.15.192:0/1494452924 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe64801d7f0 con 0x7fe6540f4850 2024-02-20T13:48:13.167 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:13.168 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:13.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 msgr2=0x7fe63c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:13.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 0x7fe63c033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe6440096a0 tx=0x7fe644006b40).stop 2024-02-20T13:48:13.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 msgr2=0x7fe6541067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:13.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fe6480089f0 tx=0x7fe64800b040).stop 2024-02-20T13:48:13.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 shutdown_connections 2024-02-20T13:48:13.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe63c031090 0x7fe63c033550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:13.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 --2- 172.21.15.192:0/1494452924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6540f4850 0x7fe6541067a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:13.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 >> 172.21.15.192:0/1494452924 conn(0x7fe6540f0390 msgr2=0x7fe6540f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:13.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 shutdown_connections 2024-02-20T13:48:13.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:13.182+0000 7fe6598d5700 1 -- 172.21.15.192:0/1494452924 wait complete. 2024-02-20T13:48:13.171 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:13.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:13.181708+0000 mon.smithi158 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.192:0/1494452924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:14.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:14.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:15.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:13.393668+0000 mgr.smithi158.gmoobf (mgr.14156) 227 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:16.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1595970486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 msgr2=0x7fa1d80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:16.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1595970486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d80f56f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa1d4009fb0 tx=0x7fa1d4009b30).stop 2024-02-20T13:48:16.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1595970486 shutdown_connections 2024-02-20T13:48:16.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1595970486 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d80f56f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1595970486 >> 172.21.15.192:0/1595970486 conn(0x7fa1d80f0a30 msgr2=0x7fa1d80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1595970486 shutdown_connections 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1595970486 wait complete. 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 Processor -- start 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.567+0000 7fa1de2b1700 1 -- start start 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1de2b1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1de2b1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d8106ae0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1dd2af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1dd2af700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58652/0 (socket says 172.21.15.192:58652) 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1dd2af700 1 -- 172.21.15.192:0/1052062376 learned_addr learned my addr 172.21.15.192:0/1052062376 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1dd2af700 1 -- 172.21.15.192:0/1052062376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4005040 con 0x7fa1d80f52d0 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1dd2af700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fa1d40089f0 tx=0x7fa1d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:16.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1d4009320 con 0x7fa1d80f52d0 2024-02-20T13:48:16.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.568+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d8106ce0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d8107170 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa1d4013ec0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1d4014d00 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa1d40263a0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1cdffb700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 0x7fa1c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.569+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa1d4042410 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.570+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d80591e0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.570+0000 7fa1dcaae700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 0x7fa1c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.570+0000 7fa1dcaae700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 0x7fa1c4033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa1c80096a0 tx=0x7fa1c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:16.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.576+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa1d4012920 con 0x7fa1d80f52d0 2024-02-20T13:48:16.767 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.781+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1d80fbef0 con 0x7fa1d80f52d0 2024-02-20T13:48:16.768 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.782+0000 7fa1cdffb700 1 -- 172.21.15.192:0/1052062376 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa1d402b020 con 0x7fa1d80f52d0 2024-02-20T13:48:16.769 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:16.769 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 msgr2=0x7fa1c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 0x7fa1c4033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa1c80096a0 tx=0x7fa1c8006b40).stop 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 msgr2=0x7fa1d81065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fa1d40089f0 tx=0x7fa1d400b040).stop 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 shutdown_connections 2024-02-20T13:48:16.770 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa1c4031090 0x7fa1c4033550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 --2- 172.21.15.192:0/1052062376 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1d80f52d0 0x7fa1d81065f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 >> 172.21.15.192:0/1052062376 conn(0x7fa1d80f0a30 msgr2=0x7fa1d80f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 shutdown_connections 2024-02-20T13:48:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:16.784+0000 7fa1de2b1700 1 -- 172.21.15.192:0/1052062376 wait complete. 2024-02-20T13:48:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:17.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:15.394064+0000 mgr.smithi158.gmoobf (mgr.14156) 228 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:17.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:16.782562+0000 mon.smithi158 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.192:0/1052062376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:18.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:18.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:19.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:17.394432+0000 mgr.smithi158.gmoobf (mgr.14156) 229 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:20.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- 172.21.15.192:0/1290800742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 msgr2=0x7f2dc80f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:20.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/1290800742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc80f9160 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f2db0009fb0 tx=0x7f2db0009b30).stop 2024-02-20T13:48:20.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- 172.21.15.192:0/1290800742 shutdown_connections 2024-02-20T13:48:20.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/1290800742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc80f9160 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:20.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- 172.21.15.192:0/1290800742 >> 172.21.15.192:0/1290800742 conn(0x7f2dc80f0be0 msgr2=0x7f2dc80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- 172.21.15.192:0/1290800742 shutdown_connections 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- 172.21.15.192:0/1290800742 wait complete. 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 Processor -- start 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.597+0000 7f2dce0f1700 1 -- start start 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dce0f1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dce0f1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8106c40 con 0x7f2dc8060360 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dc77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dc77fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58674/0 (socket says 172.21.15.192:58674) 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dc77fe700 1 -- 172.21.15.192:0/2752088007 learned_addr learned my addr 172.21.15.192:0/2752088007 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dc77fe700 1 -- 172.21.15.192:0/2752088007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0005040 con 0x7f2dc8060360 2024-02-20T13:48:20.586 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dc77fe700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2db0009f80 tx=0x7f2db000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2db0009320 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc8106e40 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2db0013680 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.598+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2db001eda0 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.599+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc81072d0 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.599+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2db0014420 con 0x7f2dc8060360 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.599+0000 7f2dbffff700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 0x7f2db4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:20.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.599+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2db0042400 con 0x7f2dc8060360 2024-02-20T13:48:20.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.600+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80591e0 con 0x7f2dc8060360 2024-02-20T13:48:20.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.600+0000 7f2dc6ffd700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 0x7f2db4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:20.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.600+0000 7f2dc6ffd700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 0x7f2db4033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2db80096a0 tx=0x7f2db8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:20.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.604+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2db0014700 con 0x7f2dc8060360 2024-02-20T13:48:20.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.811+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2dc8045bb0 con 0x7f2dc8060360 2024-02-20T13:48:20.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.812+0000 7f2dbffff700 1 -- 172.21.15.192:0/2752088007 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2db001c320 con 0x7f2dc8060360 2024-02-20T13:48:20.798 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:20.798 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:20.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 msgr2=0x7f2db4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:20.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 0x7f2db4033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2db80096a0 tx=0x7f2db8006b40).stop 2024-02-20T13:48:20.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 msgr2=0x7f2dc8106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2db0009f80 tx=0x7f2db000b040).stop 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 shutdown_connections 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2db4031090 0x7f2db4033550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 --2- 172.21.15.192:0/2752088007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dc8060360 0x7f2dc8106750 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 >> 172.21.15.192:0/2752088007 conn(0x7f2dc80f0be0 msgr2=0x7f2dc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 shutdown_connections 2024-02-20T13:48:20.800 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:20.813+0000 7f2dce0f1700 1 -- 172.21.15.192:0/2752088007 wait complete. 2024-02-20T13:48:20.801 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:21.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:19.394746+0000 mgr.smithi158.gmoobf (mgr.14156) 230 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:21.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:20.812164+0000 mon.smithi158 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.192:0/2752088007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:22.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:22.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:23.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:21.395051+0000 mgr.smithi158.gmoobf (mgr.14156) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.457+0000 7f956f9a6700 1 -- 172.21.15.192:0/2533486920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 msgr2=0x7f95680f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.457+0000 7f956f9a6700 1 --2- 172.21.15.192:0/2533486920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95680f69b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9558004760 tx=0x7f9558009ad0).stop 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.457+0000 7f956f9a6700 1 -- 172.21.15.192:0/2533486920 shutdown_connections 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.457+0000 7f956f9a6700 1 --2- 172.21.15.192:0/2533486920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95680f69b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.457+0000 7f956f9a6700 1 -- 172.21.15.192:0/2533486920 >> 172.21.15.192:0/2533486920 conn(0x7f95680f1cd0 msgr2=0x7f95680f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.458+0000 7f956f9a6700 1 -- 172.21.15.192:0/2533486920 shutdown_connections 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.458+0000 7f956f9a6700 1 -- 172.21.15.192:0/2533486920 wait complete. 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.458+0000 7f956f9a6700 1 Processor -- start 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.458+0000 7f956f9a6700 1 -- start start 2024-02-20T13:48:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956f9a6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956f9a6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9568106cb0 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956d742700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956d742700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56798/0 (socket says 172.21.15.192:56798) 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956d742700 1 -- 172.21.15.192:0/1414965623 learned_addr learned my addr 172.21.15.192:0/1414965623 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956d742700 1 -- 172.21.15.192:0/1414965623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558009d70 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956d742700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9558004170 tx=0x7f9558009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955800e040 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9568106eb0 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9558008ef0 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.459+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9558014c90 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.460+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9568107340 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.460+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9558014420 con 0x7f95680f6590 2024-02-20T13:48:24.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.460+0000 7f955dffb700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 0x7f95540335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:24.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.461+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9568045bb0 con 0x7f95680f6590 2024-02-20T13:48:24.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.461+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9558042280 con 0x7f95680f6590 2024-02-20T13:48:24.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.461+0000 7f956cf41700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 0x7f95540335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:24.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.461+0000 7f956cf41700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 0x7f95540335a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f95640096a0 tx=0x7f9564006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:24.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.466+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9558026300 con 0x7f95680f6590 2024-02-20T13:48:24.658 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.672+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9568108140 con 0x7f95680f6590 2024-02-20T13:48:24.659 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.673+0000 7f955dffb700 1 -- 172.21.15.192:0/1414965623 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9558012b20 con 0x7f95680f6590 2024-02-20T13:48:24.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:24.659 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 msgr2=0x7f95540335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 0x7f95540335a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f95640096a0 tx=0x7f9564006b40).stop 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 msgr2=0x7f95681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9558004170 tx=0x7f9558009ad0).stop 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 shutdown_connections 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f95540310e0 0x7f95540335a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 --2- 172.21.15.192:0/1414965623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f95680f6590 0x7f95681067c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 >> 172.21.15.192:0/1414965623 conn(0x7f95680f1cd0 msgr2=0x7f95680f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 shutdown_connections 2024-02-20T13:48:24.661 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:24.675+0000 7f956f9a6700 1 -- 172.21.15.192:0/1414965623 wait complete. 2024-02-20T13:48:24.662 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:23.395310+0000 mgr.smithi158.gmoobf (mgr.14156) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:25.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:24.673600+0000 mon.smithi158 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.192:0/1414965623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:26.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:26.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:25.395639+0000 mgr.smithi158.gmoobf (mgr.14156) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:28.293 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 -- 172.21.15.192:0/317085686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 msgr2=0x7f8e200f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 --2- 172.21.15.192:0/317085686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f91c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f8e10009fb0 tx=0x7f8e10009b30).stop 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 -- 172.21.15.192:0/317085686 shutdown_connections 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 --2- 172.21.15.192:0/317085686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f91c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 -- 172.21.15.192:0/317085686 >> 172.21.15.192:0/317085686 conn(0x7f8e200f0be0 msgr2=0x7f8e200f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 -- 172.21.15.192:0/317085686 shutdown_connections 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.306+0000 7f8e261e3700 1 -- 172.21.15.192:0/317085686 wait complete. 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e261e3700 1 Processor -- start 2024-02-20T13:48:28.294 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e261e3700 1 -- start start 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e261e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e261e3700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200f50c0 con 0x7f8e20060300 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e1f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e1f7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56804/0 (socket says 172.21.15.192:56804) 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.307+0000 7f8e1f7fe700 1 -- 172.21.15.192:0/810481622 learned_addr learned my addr 172.21.15.192:0/810481622 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e1f7fe700 1 -- 172.21.15.192:0/810481622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10005040 con 0x7f8e20060300 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e1f7fe700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f8e10009f80 tx=0x7f8e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e10009320 con 0x7f8e20060300 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e200f5260 con 0x7f8e20060300 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e10013680 con 0x7f8e20060300 2024-02-20T13:48:28.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e1001eda0 con 0x7f8e20060300 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.308+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e200f56f0 con 0x7f8e20060300 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.309+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8e1001e420 con 0x7f8e20060300 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.309+0000 7f8e17fff700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 0x7f8e0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.309+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e200591e0 con 0x7f8e20060300 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.309+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e10042340 con 0x7f8e20060300 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.309+0000 7f8e1effd700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 0x7f8e0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:28.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.310+0000 7f8e1effd700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 0x7f8e0c033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8e080096a0 tx=0x7f8e08006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:28.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.314+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e1001d380 con 0x7f8e20060300 2024-02-20T13:48:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.519+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e200fc060 con 0x7f8e20060300 2024-02-20T13:48:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.520+0000 7f8e17fff700 1 -- 172.21.15.192:0/810481622 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e10004490 con 0x7f8e20060300 2024-02-20T13:48:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:28.507 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 msgr2=0x7f8e0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 0x7f8e0c033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8e080096a0 tx=0x7f8e08006b40).stop 2024-02-20T13:48:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 msgr2=0x7f8e200f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f8e10009f80 tx=0x7f8e1000b040).stop 2024-02-20T13:48:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 shutdown_connections 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8e0c031090 0x7f8e0c033550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 --2- 172.21.15.192:0/810481622 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e20060300 0x7f8e200f6c40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 >> 172.21.15.192:0/810481622 conn(0x7f8e200f0be0 msgr2=0x7f8e200f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 shutdown_connections 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:28.521+0000 7f8e261e3700 1 -- 172.21.15.192:0/810481622 wait complete. 2024-02-20T13:48:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:29.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:27.395973+0000 mgr.smithi158.gmoobf (mgr.14156) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:29.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:28.521020+0000 mon.smithi158 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.192:0/810481622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:30.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:30.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:31.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:29.396235+0000 mgr.smithi158.gmoobf (mgr.14156) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:31.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 -- 172.21.15.192:0/1465060441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 msgr2=0x7fc2040f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:31.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 --2- 172.21.15.192:0/1465060441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc2040f91c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fc1f8009fb0 tx=0x7fc1f8009b30).stop 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 -- 172.21.15.192:0/1465060441 shutdown_connections 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 --2- 172.21.15.192:0/1465060441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc2040f91c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 -- 172.21.15.192:0/1465060441 >> 172.21.15.192:0/1465060441 conn(0x7fc2040f0be0 msgr2=0x7fc2040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 -- 172.21.15.192:0/1465060441 shutdown_connections 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 -- 172.21.15.192:0/1465060441 wait complete. 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.004+0000 7fc20be94700 1 Processor -- start 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc20be94700 1 -- start start 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc20be94700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc20be94700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc204106c70 con 0x7fc204060300 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc209c30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc209c30700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56816/0 (socket says 172.21.15.192:56816) 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc209c30700 1 -- 172.21.15.192:0/3320244225 learned_addr learned my addr 172.21.15.192:0/3320244225 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc209c30700 1 -- 172.21.15.192:0/3320244225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8005040 con 0x7fc204060300 2024-02-20T13:48:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc209c30700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc1f8009f80 tx=0x7fc1f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1f8009320 con 0x7fc204060300 2024-02-20T13:48:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc204106e70 con 0x7fc204060300 2024-02-20T13:48:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc1f8013680 con 0x7fc204060300 2024-02-20T13:48:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.005+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1f801eda0 con 0x7fc204060300 2024-02-20T13:48:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.006+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc204107300 con 0x7fc204060300 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.006+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc1f801e420 con 0x7fc204060300 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.006+0000 7fc1f67fc700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 0x7fc1f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.006+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1f8042400 con 0x7fc204060300 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.006+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc204045bb0 con 0x7fc204060300 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.007+0000 7fc20942f700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 0x7fc1f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.007+0000 7fc20942f700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 0x7fc1f0033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc2000096a0 tx=0x7fc200006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.011+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc1f801e700 con 0x7fc204060300 2024-02-20T13:48:32.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.218+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2040fc150 con 0x7fc204060300 2024-02-20T13:48:32.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.219+0000 7fc1f67fc700 1 -- 172.21.15.192:0/3320244225 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc1f801cb10 con 0x7fc204060300 2024-02-20T13:48:32.206 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:32.206 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 msgr2=0x7fc1f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 0x7fc1f0033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc2000096a0 tx=0x7fc200006b40).stop 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 msgr2=0x7fc204106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc1f8009f80 tx=0x7fc1f800b040).stop 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 shutdown_connections 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc1f0031090 0x7fc1f0033550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 --2- 172.21.15.192:0/3320244225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc204060300 0x7fc204106780 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 >> 172.21.15.192:0/3320244225 conn(0x7fc2040f0be0 msgr2=0x7fc2040f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 shutdown_connections 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:32.220+0000 7fc20be94700 1 -- 172.21.15.192:0/3320244225 wait complete. 2024-02-20T13:48:32.208 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:33.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:31.396568+0000 mgr.smithi158.gmoobf (mgr.14156) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:33.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:32.220319+0000 mon.smithi158 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.192:0/3320244225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:34.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:34.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:35.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:33.396976+0000 mgr.smithi158.gmoobf (mgr.14156) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 -- 172.21.15.192:0/1689626618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 msgr2=0x7f88600f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 --2- 172.21.15.192:0/1689626618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88600f5e80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f8850009fb0 tx=0x7f8850009b30).stop 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 -- 172.21.15.192:0/1689626618 shutdown_connections 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 --2- 172.21.15.192:0/1689626618 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88600f5e80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 -- 172.21.15.192:0/1689626618 >> 172.21.15.192:0/1689626618 conn(0x7f88600f0390 msgr2=0x7f88600f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 -- 172.21.15.192:0/1689626618 shutdown_connections 2024-02-20T13:48:35.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 -- 172.21.15.192:0/1689626618 wait complete. 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.975+0000 7f8865348700 1 Processor -- start 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f8865348700 1 -- start start 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f8865348700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f8865348700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88600f5420 con 0x7f88600f5a60 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f885effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f885effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53262/0 (socket says 172.21.15.192:53262) 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f885effd700 1 -- 172.21.15.192:0/3279510559 learned_addr learned my addr 172.21.15.192:0/3279510559 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f885effd700 1 -- 172.21.15.192:0/3279510559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8850005040 con 0x7f88600f5a60 2024-02-20T13:48:35.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.976+0000 7f885effd700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f8850000c00 tx=0x7f885000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:35.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.977+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88500095f0 con 0x7f88600f5a60 2024-02-20T13:48:35.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.977+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8850007e50 con 0x7f88600f5a60 2024-02-20T13:48:35.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.977+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88600f5620 con 0x7f88600f5a60 2024-02-20T13:48:35.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.977+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8850014e70 con 0x7f88600f5a60 2024-02-20T13:48:35.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.977+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8860109ba0 con 0x7f88600f5a60 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8850025070 con 0x7f88600f5a60 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f88477fe700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 0x7f8848033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8850041e50 con 0x7f88600f5a60 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88600591e0 con 0x7f88600f5a60 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f885e7fc700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 0x7f8848033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.978+0000 7f885e7fc700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 0x7f8848033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f88540096a0 tx=0x7f8854006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:35.969 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:35.983+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8850026350 con 0x7f88600f5a60 2024-02-20T13:48:36.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.195+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f886010a1b0 con 0x7f88600f5a60 2024-02-20T13:48:36.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.195+0000 7f88477fe700 1 -- 172.21.15.192:0/3279510559 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8850025300 con 0x7f88600f5a60 2024-02-20T13:48:36.183 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:36.183 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:36.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 msgr2=0x7f8848033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:36.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 0x7f8848033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f88540096a0 tx=0x7f8854006b40).stop 2024-02-20T13:48:36.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 msgr2=0x7f88601094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:36.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f8850000c00 tx=0x7f885000b040).stop 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 shutdown_connections 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8848031090 0x7f8848033550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 --2- 172.21.15.192:0/3279510559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88600f5a60 0x7f88601094f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 >> 172.21.15.192:0/3279510559 conn(0x7f88600f0390 msgr2=0x7f88600f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 shutdown_connections 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:36.197+0000 7f8865348700 1 -- 172.21.15.192:0/3279510559 wait complete. 2024-02-20T13:48:36.185 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:37.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:35.397327+0000 mgr.smithi158.gmoobf (mgr.14156) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:37.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:36.196852+0000 mon.smithi158 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.192:0/3279510559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:37.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:37.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:39.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:37.397717+0000 mgr.smithi158.gmoobf (mgr.14156) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- 172.21.15.192:0/2055766006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 msgr2=0x7f6bac0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/2055766006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac0f58a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6ba0009fb0 tx=0x7f6ba0009b30).stop 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- 172.21.15.192:0/2055766006 shutdown_connections 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/2055766006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac0f58a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- 172.21.15.192:0/2055766006 >> 172.21.15.192:0/2055766006 conn(0x7f6bac0f0be0 msgr2=0x7f6bac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:39.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- 172.21.15.192:0/2055766006 shutdown_connections 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- 172.21.15.192:0/2055766006 wait complete. 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 Processor -- start 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- start start 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.761+0000 7f6bb11cb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac105be0 con 0x7f6bac0f5480 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6baad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6baad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53278/0 (socket says 172.21.15.192:53278) 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6baad9d700 1 -- 172.21.15.192:0/116845981 learned_addr learned my addr 172.21.15.192:0/116845981 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6baad9d700 1 -- 172.21.15.192:0/116845981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0005040 con 0x7f6bac0f5480 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6baad9d700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f6ba00089f0 tx=0x7f6ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ba0009320 con 0x7f6bac0f5480 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6ba0013ec0 con 0x7f6bac0f5480 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac105de0 con 0x7f6bac0f5480 2024-02-20T13:48:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ba0014d30 con 0x7f6bac0f5480 2024-02-20T13:48:39.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.762+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac106270 con 0x7f6bac0f5480 2024-02-20T13:48:39.751 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.763+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6ba00095f0 con 0x7f6bac0f5480 2024-02-20T13:48:39.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.763+0000 7f6b937fe700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 0x7f6b94033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:39.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.763+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac045bb0 con 0x7f6bac0f5480 2024-02-20T13:48:39.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.763+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6ba0042390 con 0x7f6bac0f5480 2024-02-20T13:48:39.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.764+0000 7f6baa59c700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 0x7f6b94033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:39.752 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.764+0000 7f6baa59c700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 0x7f6b94033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f6b9c0096a0 tx=0x7f6b9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:39.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.768+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6ba001d7f0 con 0x7f6bac0f5480 2024-02-20T13:48:39.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.977+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bac0fb060 con 0x7f6bac0f5480 2024-02-20T13:48:39.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.977+0000 7f6b937fe700 1 -- 172.21.15.192:0/116845981 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ba002c090 con 0x7f6bac0f5480 2024-02-20T13:48:39.964 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:39.964 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 msgr2=0x7f6b94033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 0x7f6b94033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f6b9c0096a0 tx=0x7f6b9c006b40).stop 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 msgr2=0x7f6bac1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f6ba00089f0 tx=0x7f6ba000b040).stop 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 shutdown_connections 2024-02-20T13:48:39.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6b94031090 0x7f6b94033550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 --2- 172.21.15.192:0/116845981 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bac0f5480 0x7f6bac1056f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 >> 172.21.15.192:0/116845981 conn(0x7f6bac0f0be0 msgr2=0x7f6bac0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 shutdown_connections 2024-02-20T13:48:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:39.979+0000 7f6bb11cb700 1 -- 172.21.15.192:0/116845981 wait complete. 2024-02-20T13:48:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:39.978901+0000 mon.smithi158 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.192:0/116845981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:41.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:39.398080+0000 mgr.smithi158.gmoobf (mgr.14156) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:41.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:41.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/3261139765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 msgr2=0x7f8fa40f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/3261139765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa40f5030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8f94009fb0 tx=0x7f8f94009b30).stop 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/3261139765 shutdown_connections 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/3261139765 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa40f5030 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/3261139765 >> 172.21.15.192:0/3261139765 conn(0x7f8fa40f0390 msgr2=0x7f8fa40f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/3261139765 shutdown_connections 2024-02-20T13:48:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/3261139765 wait complete. 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.404+0000 7f8fa9bdd700 1 Processor -- start 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa9bdd700 1 -- start start 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa9bdd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa9bdd700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa41072d0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54880/0 (socket says 172.21.15.192:54880) 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa37fe700 1 -- 172.21.15.192:0/1216812960 learned_addr learned my addr 172.21.15.192:0/1216812960 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa37fe700 1 -- 172.21.15.192:0/1216812960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94005040 con 0x7f8fa40f4c10 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.405+0000 7f8fa37fe700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8f94000c00 tx=0x7f8f9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:43.393 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.406+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f940095f0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.406+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa41074d0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.406+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8f94007e50 con 0x7f8fa40f4c10 2024-02-20T13:48:43.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.406+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f94014e70 con 0x7f8fa40f4c10 2024-02-20T13:48:43.394 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.406+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa4107960 con 0x7f8fa40f4c10 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8f94025070 con 0x7f8fa40f4c10 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8f8bfff700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 0x7f8f8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa40591e0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8f94041d10 con 0x7f8fa40f4c10 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8fa2ffd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 0x7f8f8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:43.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.407+0000 7f8fa2ffd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 0x7f8f8c033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8f980096a0 tx=0x7f8f98006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:43.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.412+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8f940079c0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:41.398380+0000 mgr.smithi158.gmoobf (mgr.14156) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:43.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.617+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fa4045bb0 con 0x7f8fa40f4c10 2024-02-20T13:48:43.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.618+0000 7f8f8bfff700 1 -- 172.21.15.192:0/1216812960 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f9400f300 con 0x7f8fa40f4c10 2024-02-20T13:48:43.605 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:43.605 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 msgr2=0x7f8f8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 0x7f8f8c033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8f980096a0 tx=0x7f8f98006b40).stop 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 msgr2=0x7f8fa4106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8f94000c00 tx=0x7f8f9400b040).stop 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 shutdown_connections 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8f8c031090 0x7f8f8c033550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 --2- 172.21.15.192:0/1216812960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fa40f4c10 0x7f8fa4106de0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 >> 172.21.15.192:0/1216812960 conn(0x7f8fa40f0390 msgr2=0x7f8fa40f0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 shutdown_connections 2024-02-20T13:48:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:43.619+0000 7f8fa9bdd700 1 -- 172.21.15.192:0/1216812960 wait complete. 2024-02-20T13:48:43.607 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:44.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:44 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:43.619065+0000 mon.smithi158 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.192:0/1216812960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:45.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:45.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:45.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:43.398635+0000 mgr.smithi158.gmoobf (mgr.14156) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 -- 172.21.15.192:0/96538164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 msgr2=0x7f2cbc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/96538164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc0f58a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f2cb0009fb0 tx=0x7f2cb0009b30).stop 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 -- 172.21.15.192:0/96538164 shutdown_connections 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/96538164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc0f58a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 -- 172.21.15.192:0/96538164 >> 172.21.15.192:0/96538164 conn(0x7f2cbc0f0be0 msgr2=0x7f2cbc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 -- 172.21.15.192:0/96538164 shutdown_connections 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 -- 172.21.15.192:0/96538164 wait complete. 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.181+0000 7f2cc2c27700 1 Processor -- start 2024-02-20T13:48:47.182 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc2c27700 1 -- start start 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc2c27700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc2c27700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc060300 con 0x7f2cbc0f5480 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc09c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc09c3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54896/0 (socket says 172.21.15.192:54896) 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc09c3700 1 -- 172.21.15.192:0/4155085471 learned_addr learned my addr 172.21.15.192:0/4155085471 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc09c3700 1 -- 172.21.15.192:0/4155085471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0005040 con 0x7f2cbc0f5480 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.182+0000 7f2cc09c3700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2cb0000c00 tx=0x7f2cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.183+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cb00095f0 con 0x7f2cbc0f5480 2024-02-20T13:48:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.183+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc060500 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.183+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2cb0007e50 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.183+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cb0014e70 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.183+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc109bc0 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2cb0025070 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cb97fa700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 0x7f2cac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2cb0041d60 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc0591e0 con 0x7f2cbc0f5480 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cbbfff700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 0x7f2cac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.184+0000 7f2cbbfff700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 0x7f2cac033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2ca80096a0 tx=0x7f2ca8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:47.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.189+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2cb0026350 con 0x7f2cbc0f5480 2024-02-20T13:48:47.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.394+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cbc10a290 con 0x7f2cbc0f5480 2024-02-20T13:48:47.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.395+0000 7f2cb97fa700 1 -- 172.21.15.192:0/4155085471 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2cb0012330 con 0x7f2cbc0f5480 2024-02-20T13:48:47.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:47.382 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:47.383 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 msgr2=0x7f2cac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:47.383 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 0x7f2cac033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2ca80096a0 tx=0x7f2ca8006b40).stop 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 msgr2=0x7f2cbc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2cb0000c00 tx=0x7f2cb000b040).stop 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 shutdown_connections 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2cac031090 0x7f2cac033550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 --2- 172.21.15.192:0/4155085471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2cbc0f5480 0x7f2cbc109510 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 >> 172.21.15.192:0/4155085471 conn(0x7f2cbc0f0be0 msgr2=0x7f2cbc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 shutdown_connections 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:47.396+0000 7f2cc2c27700 1 -- 172.21.15.192:0/4155085471 wait complete. 2024-02-20T13:48:47.384 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:45.399030+0000 mgr.smithi158.gmoobf (mgr.14156) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:48:45.836874+0000 mgr.smithi158.gmoobf (mgr.14156) 244 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon node-exporter.smithi158 ... 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Verifying port 9100 ... 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 1 from systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr Job for ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service failed because a timeout was exceeded. 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr See "systemctl status ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service" and "journalctl -xe" for details. 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ports=daemon_ports) 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:48:47.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: Failed command: systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: yield (conn, connr) 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: code, '\n'.join(err))) 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon node-exporter.smithi158 ... 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Verifying port 9100 ... 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 1 from systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr Job for ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service failed because a timeout was exceeded. 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr See "systemctl status ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service" and "journalctl -xe" for details. 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ports=daemon_ports) 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:48:47.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: Failed command: systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:48:45.837078+0000 mgr.smithi158.gmoobf (mgr.14156) 245 : cephadm [ERR] Failed while placing node-exporter.smithi158 on smithi158: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-node-exporter.smithi158 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon node-exporter.smithi158 ... 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Verifying port 9100 ... 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 1 from systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr Job for ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service failed because a timeout was exceeded. 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: systemctl: stderr See "systemctl status ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158.service" and "journalctl -xe" for details. 2024-02-20T13:48:47.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: ports=daemon_ports) 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:48:47.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:48:47.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: Failed command: systemctl start ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@node-exporter.smithi158 2024-02-20T13:48:47.540 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:48:45.843292+0000 mgr.smithi158.gmoobf (mgr.14156) 246 : cephadm [INF] Deploying daemon prometheus.smithi158 on smithi158 2024-02-20T13:48:48.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:47.396175+0000 mon.smithi158 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.192:0/4155085471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:49.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:49.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:49.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:47.399312+0000 mgr.smithi158.gmoobf (mgr.14156) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:51.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 -- 172.21.15.192:0/767701426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 msgr2=0x7f39fc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 --2- 172.21.15.192:0/767701426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc0f56f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f39f0009fb0 tx=0x7f39f0009b30).stop 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 -- 172.21.15.192:0/767701426 shutdown_connections 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 --2- 172.21.15.192:0/767701426 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc0f56f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 -- 172.21.15.192:0/767701426 >> 172.21.15.192:0/767701426 conn(0x7f39fc0f0a30 msgr2=0x7f39fc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.012+0000 7f3a00a34700 1 -- 172.21.15.192:0/767701426 shutdown_connections 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.013+0000 7f3a00a34700 1 -- 172.21.15.192:0/767701426 wait complete. 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.013+0000 7f3a00a34700 1 Processor -- start 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.013+0000 7f3a00a34700 1 -- start start 2024-02-20T13:48:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.013+0000 7f3a00a34700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.013+0000 7f3a00a34700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc105a60 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39fad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39fad9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:54910/0 (socket says 172.21.15.192:54910) 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39fad9d700 1 -- 172.21.15.192:0/113986735 learned_addr learned my addr 172.21.15.192:0/113986735 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39fad9d700 1 -- 172.21.15.192:0/113986735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f0005040 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39fad9d700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f39f00089f0 tx=0x7f39f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39f0009320 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39fc105c60 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f39f0013ec0 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39f0014d30 con 0x7f39fc0f52d0 2024-02-20T13:48:51.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.014+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39fc1060f0 con 0x7f39fc0f52d0 2024-02-20T13:48:51.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.015+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f39f0014420 con 0x7f39fc0f52d0 2024-02-20T13:48:51.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.015+0000 7f39e37fe700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 0x7f39e40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:51.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.015+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f39f0044270 con 0x7f39fc0f52d0 2024-02-20T13:48:51.004 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.015+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39fc045bb0 con 0x7f39fc0f52d0 2024-02-20T13:48:51.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.016+0000 7f39fa59c700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 0x7f39e40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:51.005 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.016+0000 7f39fa59c700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 0x7f39e40335a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f39ec0096a0 tx=0x7f39ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:51.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.020+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f39f0014750 con 0x7f39fc0f52d0 2024-02-20T13:48:51.213 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.226+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39fc106ef0 con 0x7f39fc0f52d0 2024-02-20T13:48:51.214 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.227+0000 7f39e37fe700 1 -- 172.21.15.192:0/113986735 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39f0027e80 con 0x7f39fc0f52d0 2024-02-20T13:48:51.214 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:51.214 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 msgr2=0x7f39e40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 0x7f39e40335a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f39ec0096a0 tx=0x7f39ec006b40).stop 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 msgr2=0x7f39fc105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f39f00089f0 tx=0x7f39f000b040).stop 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 shutdown_connections 2024-02-20T13:48:51.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f39e40310e0 0x7f39e40335a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:51.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 --2- 172.21.15.192:0/113986735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39fc0f52d0 0x7f39fc105570 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:51.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 >> 172.21.15.192:0/113986735 conn(0x7f39fc0f0a30 msgr2=0x7f39fc0f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:51.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 shutdown_connections 2024-02-20T13:48:51.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:51.229+0000 7f3a00a34700 1 -- 172.21.15.192:0/113986735 wait complete. 2024-02-20T13:48:51.217 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:51.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:49.399688+0000 mgr.smithi158.gmoobf (mgr.14156) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:51.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:50.469657+0000 mon.smithi158 (mon.0) 252 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:48:52.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:52 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:51.228362+0000 mon.smithi158 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.192:0/113986735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:52.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:52.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:53.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:51.400067+0000 mgr.smithi158.gmoobf (mgr.14156) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 -- 172.21.15.192:0/103333220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 msgr2=0x7fe8100f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 --2- 172.21.15.192:0/103333220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8100f4c60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fe800009fb0 tx=0x7fe800009b30).stop 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 -- 172.21.15.192:0/103333220 shutdown_connections 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 --2- 172.21.15.192:0/103333220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8100f4c60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 -- 172.21.15.192:0/103333220 >> 172.21.15.192:0/103333220 conn(0x7fe8100f0360 msgr2=0x7fe8100f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 -- 172.21.15.192:0/103333220 shutdown_connections 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.890+0000 7fe814f0a700 1 -- 172.21.15.192:0/103333220 wait complete. 2024-02-20T13:48:54.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe814f0a700 1 Processor -- start 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe814f0a700 1 -- start start 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe814f0a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe814f0a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe810106cb0 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe80e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe80e59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:49440/0 (socket says 172.21.15.192:49440) 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe80e59c700 1 -- 172.21.15.192:0/935222758 learned_addr learned my addr 172.21.15.192:0/935222758 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe80e59c700 1 -- 172.21.15.192:0/935222758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800005040 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe80e59c700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe8000089f0 tx=0x7fe80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe800009320 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.891+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe810106eb0 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe800013ec0 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe800014d30 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe810107340 con 0x7fe8100f4840 2024-02-20T13:48:54.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fe8000095f0 con 0x7fe8100f4840 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe810045bb0 con 0x7fe8100f4840 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe7feffd700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 0x7fe7f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.892+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8000423c0 con 0x7fe8100f4840 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.893+0000 7fe80dd9b700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 0x7fe7f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.893+0000 7fe80dd9b700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 0x7fe7f8033550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe8040096a0 tx=0x7fe804006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:54.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:54.897+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe80001d980 con 0x7fe8100f4840 2024-02-20T13:48:55.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.106+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8101081f0 con 0x7fe8100f4840 2024-02-20T13:48:55.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.107+0000 7fe7feffd700 1 -- 172.21.15.192:0/935222758 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe80001d380 con 0x7fe8100f4840 2024-02-20T13:48:55.094 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:55.094 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 msgr2=0x7fe7f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 0x7fe7f8033550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe8040096a0 tx=0x7fe804006b40).stop 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 msgr2=0x7fe8101067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe8000089f0 tx=0x7fe80000b040).stop 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 shutdown_connections 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fe7f8031090 0x7fe7f8033550 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 --2- 172.21.15.192:0/935222758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8100f4840 0x7fe8101067c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 >> 172.21.15.192:0/935222758 conn(0x7fe8100f0360 msgr2=0x7fe8100f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:55.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 shutdown_connections 2024-02-20T13:48:55.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:55.108+0000 7fe814f0a700 1 -- 172.21.15.192:0/935222758 wait complete. 2024-02-20T13:48:55.096 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:55.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:53.400414+0000 mgr.smithi158.gmoobf (mgr.14156) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:55.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:55.108160+0000 mon.smithi158 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.192:0/935222758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:48:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:55.466431+0000 mon.smithi158 (mon.0) 255 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:48:56.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:55.466633+0000 mon.smithi158 (mon.0) 256 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:48:56.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:48:56.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:48:57.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:55.400830+0000 mgr.smithi158.gmoobf (mgr.14156) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/1427926476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 msgr2=0x7f3d840f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/1427926476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d840f58a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3d74009fb0 tx=0x7f3d74009b30).stop 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/1427926476 shutdown_connections 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/1427926476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d840f58a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/1427926476 >> 172.21.15.192:0/1427926476 conn(0x7f3d840f0be0 msgr2=0x7f3d840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/1427926476 shutdown_connections 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.718+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/1427926476 wait complete. 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d8a7b5700 1 Processor -- start 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d8a7b5700 1 -- start start 2024-02-20T13:48:58.730 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d8a7b5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d8a7b5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d84060360 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d83fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d83fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:49456/0 (socket says 172.21.15.192:49456) 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d83fff700 1 -- 172.21.15.192:0/3033847133 learned_addr learned my addr 172.21.15.192:0/3033847133 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.719+0000 7f3d83fff700 1 -- 172.21.15.192:0/3033847133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74005040 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d83fff700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f3d740089f0 tx=0x7f3d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74009320 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d84060560 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3d74013ec0 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74014d30 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.720+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d84109be0 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3d74014420 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d84045bb0 con 0x7f3d840f5480 2024-02-20T13:48:58.731 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d80ff9700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 0x7f3d70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:48:58.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3d74042490 con 0x7f3d840f5480 2024-02-20T13:48:58.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d837fe700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 0x7f3d70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:48:58.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.721+0000 7f3d837fe700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 0x7f3d70033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3d6c0096a0 tx=0x7f3d6c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:48:58.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.726+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3d7401d7f0 con 0x7f3d840f5480 2024-02-20T13:48:58.918 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.931+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d8410a400 con 0x7f3d840f5480 2024-02-20T13:48:58.918 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.932+0000 7f3d80ff9700 1 -- 172.21.15.192:0/3033847133 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d7400fe40 con 0x7f3d840f5480 2024-02-20T13:48:58.919 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:48:58.919 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:48:58.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 msgr2=0x7f3d70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 0x7f3d70033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3d6c0096a0 tx=0x7f3d6c006b40).stop 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 msgr2=0x7f3d841094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f3d740089f0 tx=0x7f3d7400b040).stop 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 shutdown_connections 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3d70031090 0x7f3d70033550 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 --2- 172.21.15.192:0/3033847133 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d840f5480 0x7f3d841094b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 >> 172.21.15.192:0/3033847133 conn(0x7f3d840f0be0 msgr2=0x7f3d840f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 shutdown_connections 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:48:58.933+0000 7f3d8a7b5700 1 -- 172.21.15.192:0/3033847133 wait complete. 2024-02-20T13:48:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:48:59.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:57.401122+0000 mgr.smithi158.gmoobf (mgr.14156) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:48:59.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:48:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:48:58.933071+0000 mon.smithi158 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.192:0/3033847133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:00.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:00.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:01.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:48:59.401469+0000 mgr.smithi158.gmoobf (mgr.14156) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.497+0000 7f22c9455700 1 -- 172.21.15.192:0/1431178905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 msgr2=0x7f22c40f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.497+0000 7f22c9455700 1 --2- 172.21.15.192:0/1431178905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c40f4c70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f22ac009fb0 tx=0x7f22ac009b30).stop 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- 172.21.15.192:0/1431178905 shutdown_connections 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 --2- 172.21.15.192:0/1431178905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c40f4c70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- 172.21.15.192:0/1431178905 >> 172.21.15.192:0/1431178905 conn(0x7f22c40f0390 msgr2=0x7f22c40f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- 172.21.15.192:0/1431178905 shutdown_connections 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- 172.21.15.192:0/1431178905 wait complete. 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 Processor -- start 2024-02-20T13:49:02.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- start start 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.498+0000 7f22c9455700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c4106c70 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c2ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58866/0 (socket says 172.21.15.192:58866) 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c2ffd700 1 -- 172.21.15.192:0/782637295 learned_addr learned my addr 172.21.15.192:0/782637295 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c2ffd700 1 -- 172.21.15.192:0/782637295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac005040 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c2ffd700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f22ac0089f0 tx=0x7f22ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac009320 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c4106e70 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22ac013ec0 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac014d30 con 0x7f22c40f4850 2024-02-20T13:49:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.499+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c4107300 con 0x7f22c40f4850 2024-02-20T13:49:02.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.500+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f22ac0263a0 con 0x7f22c40f4850 2024-02-20T13:49:02.489 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.500+0000 7f22bb7fe700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 0x7f22b00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:02.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.500+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f22ac042470 con 0x7f22c40f4850 2024-02-20T13:49:02.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.500+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c40591e0 con 0x7f22c40f4850 2024-02-20T13:49:02.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.500+0000 7f22c27fc700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 0x7f22b00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:02.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.501+0000 7f22c27fc700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 0x7f22b00335a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f22b40096a0 tx=0x7f22b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:02.492 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.505+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f22ac0266d0 con 0x7f22c40f4850 2024-02-20T13:49:02.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.712+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22c4045bb0 con 0x7f22c40f4850 2024-02-20T13:49:02.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.712+0000 7f22bb7fe700 1 -- 172.21.15.192:0/782637295 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22ac02b020 con 0x7f22c40f4850 2024-02-20T13:49:02.699 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:02.699 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 msgr2=0x7f22b00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 0x7f22b00335a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f22b40096a0 tx=0x7f22b4006b40).stop 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 msgr2=0x7f22c4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f22ac0089f0 tx=0x7f22ac00b040).stop 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 shutdown_connections 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f22b00310e0 0x7f22b00335a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 --2- 172.21.15.192:0/782637295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22c40f4850 0x7f22c4106780 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:02.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 >> 172.21.15.192:0/782637295 conn(0x7f22c40f0390 msgr2=0x7f22c40f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:02.702 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 shutdown_connections 2024-02-20T13:49:02.702 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:02.714+0000 7f22c9455700 1 -- 172.21.15.192:0/782637295 wait complete. 2024-02-20T13:49:02.702 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:03.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:01.401904+0000 mgr.smithi158.gmoobf (mgr.14156) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:03.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:02.713833+0000 mon.smithi158 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.192:0/782637295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:04.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:04.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:05.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:03.402287+0000 mgr.smithi158.gmoobf (mgr.14156) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:06.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 -- 172.21.15.192:0/961632994 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 msgr2=0x7ff86c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 --2- 172.21.15.192:0/961632994 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c0f69b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7ff85c004760 tx=0x7ff85c009ad0).stop 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 -- 172.21.15.192:0/961632994 shutdown_connections 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 --2- 172.21.15.192:0/961632994 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c0f69b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 -- 172.21.15.192:0/961632994 >> 172.21.15.192:0/961632994 conn(0x7ff86c0f1cd0 msgr2=0x7ff86c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 -- 172.21.15.192:0/961632994 shutdown_connections 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.144+0000 7ff8744ba700 1 -- 172.21.15.192:0/961632994 wait complete. 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff8744ba700 1 Processor -- start 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff8744ba700 1 -- start start 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff8744ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff8744ba700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c106cb0 con 0x7ff86c0f6590 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff872256700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff872256700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58896/0 (socket says 172.21.15.192:58896) 2024-02-20T13:49:06.135 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff872256700 1 -- 172.21.15.192:0/3765685188 learned_addr learned my addr 172.21.15.192:0/3765685188 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.145+0000 7ff872256700 1 -- 172.21.15.192:0/3765685188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c009d70 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff872256700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff85c013f10 tx=0x7ff85c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff85c00e040 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c106eb0 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff85c007df0 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff85c014d60 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.146+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c107340 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff85c014520 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c045bb0 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff862ffd700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff85c042490 con 0x7ff86c0f6590 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff871a55700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.147+0000 7ff871a55700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 0x7ff858033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7ff8680096a0 tx=0x7ff868006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:06.138 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.153+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff85c007960 con 0x7ff86c0f6590 2024-02-20T13:49:06.343 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.358+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff86c1081f0 con 0x7ff86c0f6590 2024-02-20T13:49:06.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.358+0000 7ff862ffd700 1 -- 172.21.15.192:0/3765685188 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff85c0262f0 con 0x7ff86c0f6590 2024-02-20T13:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:06.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 msgr2=0x7ff858033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:06.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 0x7ff858033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7ff8680096a0 tx=0x7ff868006b40).stop 2024-02-20T13:49:06.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 msgr2=0x7ff86c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff85c013f10 tx=0x7ff85c009ad0).stop 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 shutdown_connections 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 --2- 172.21.15.192:0/3765685188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff86c0f6590 0x7ff86c1067c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 >> 172.21.15.192:0/3765685188 conn(0x7ff86c0f1cd0 msgr2=0x7ff86c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:06.347 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 shutdown_connections 2024-02-20T13:49:06.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:06.360+0000 7ff8744ba700 1 -- 172.21.15.192:0/3765685188 wait complete. 2024-02-20T13:49:06.348 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:07.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:05.402646+0000 mgr.smithi158.gmoobf (mgr.14156) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:07.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:06.358935+0000 mon.smithi158 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.192:0/3765685188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:08.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:08.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:09.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:07.402942+0000 mgr.smithi158.gmoobf (mgr.14156) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 -- 172.21.15.192:0/3074624976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 msgr2=0x7ff1900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 --2- 172.21.15.192:0/3074624976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1900f58a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff178009fb0 tx=0x7ff178009b30).stop 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 -- 172.21.15.192:0/3074624976 shutdown_connections 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 --2- 172.21.15.192:0/3074624976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1900f58a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 -- 172.21.15.192:0/3074624976 >> 172.21.15.192:0/3074624976 conn(0x7ff1900f0be0 msgr2=0x7ff1900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 -- 172.21.15.192:0/3074624976 shutdown_connections 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.085+0000 7ff19636a700 1 -- 172.21.15.192:0/3074624976 wait complete. 2024-02-20T13:49:10.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.086+0000 7ff19636a700 1 Processor -- start 2024-02-20T13:49:10.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.086+0000 7ff19636a700 1 -- start start 2024-02-20T13:49:10.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.086+0000 7ff19636a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:10.073 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.086+0000 7ff19636a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190105be0 con 0x7ff1900f5480 2024-02-20T13:49:10.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:10.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58920/0 (socket says 172.21.15.192:58920) 2024-02-20T13:49:10.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18ffff700 1 -- 172.21.15.192:0/3381285440 learned_addr learned my addr 172.21.15.192:0/3381285440 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:10.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18ffff700 1 -- 172.21.15.192:0/3381285440 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178005040 con 0x7ff1900f5480 2024-02-20T13:49:10.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18ffff700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ff1780089f0 tx=0x7ff17800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff178009320 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff190105de0 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff178013ec0 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff178014d30 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.087+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff190106270 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.088+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff178014420 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.088+0000 7ff18cff9700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 0x7ff17c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.088+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff178042350 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.088+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff190045bb0 con 0x7ff1900f5480 2024-02-20T13:49:10.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.089+0000 7ff18f7fe700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 0x7ff17c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:10.076 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.089+0000 7ff18f7fe700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 0x7ff17c0335a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff1800096a0 tx=0x7ff180006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:10.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.094+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff178014750 con 0x7ff1900f5480 2024-02-20T13:49:10.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.300+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1900fb060 con 0x7ff1900f5480 2024-02-20T13:49:10.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.300+0000 7ff18cff9700 1 -- 172.21.15.192:0/3381285440 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff17801d7f0 con 0x7ff1900f5480 2024-02-20T13:49:10.286 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:10.286 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:10.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 msgr2=0x7ff17c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:10.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 0x7ff17c0335a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff1800096a0 tx=0x7ff180006b40).stop 2024-02-20T13:49:10.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 msgr2=0x7ff1901056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:10.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ff1780089f0 tx=0x7ff17800b040).stop 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 shutdown_connections 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7ff17c0310e0 0x7ff17c0335a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 --2- 172.21.15.192:0/3381285440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1900f5480 0x7ff1901056f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 >> 172.21.15.192:0/3381285440 conn(0x7ff1900f0be0 msgr2=0x7ff1900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 shutdown_connections 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:10.302+0000 7ff19636a700 1 -- 172.21.15.192:0/3381285440 wait complete. 2024-02-20T13:49:10.289 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:11.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:09.403211+0000 mgr.smithi158.gmoobf (mgr.14156) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:11.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:10.300764+0000 mon.smithi158 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.192:0/3381285440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:12.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:12.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:13.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:11.403561+0000 mgr.smithi158.gmoobf (mgr.14156) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:13.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 -- 172.21.15.192:0/1588236795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 msgr2=0x7f64800f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 --2- 172.21.15.192:0/1588236795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64800f4c70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f6470009fb0 tx=0x7f6470009b30).stop 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 -- 172.21.15.192:0/1588236795 shutdown_connections 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 --2- 172.21.15.192:0/1588236795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64800f4c70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 -- 172.21.15.192:0/1588236795 >> 172.21.15.192:0/1588236795 conn(0x7f64800f0390 msgr2=0x7f64800f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 -- 172.21.15.192:0/1588236795 shutdown_connections 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.974+0000 7f64877bf700 1 -- 172.21.15.192:0/1588236795 wait complete. 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f64877bf700 1 Processor -- start 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f64877bf700 1 -- start start 2024-02-20T13:49:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f64877bf700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f64877bf700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480106ca0 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f648555b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f648555b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51264/0 (socket says 172.21.15.192:51264) 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f648555b700 1 -- 172.21.15.192:0/4168622182 learned_addr learned my addr 172.21.15.192:0/4168622182 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f648555b700 1 -- 172.21.15.192:0/4168622182 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470005040 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.975+0000 7f648555b700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f64700089f0 tx=0x7f647000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.976+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6470009320 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.976+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6480106ea0 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.976+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6470013ec0 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.976+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6470014d30 con 0x7f64800f4850 2024-02-20T13:49:13.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.976+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6480107330 con 0x7f64800f4850 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f64700095f0 con 0x7f64800f4850 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480045bb0 con 0x7f64800f4850 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f6475ffb700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6470042390 con 0x7f64800f4850 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f6484d5a700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:13.964 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.977+0000 7f6484d5a700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 0x7f646c033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f647c0096a0 tx=0x7f647c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:13.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:13.982+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f647001d7f0 con 0x7f64800f4850 2024-02-20T13:49:14.173 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.187+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64801081e0 con 0x7f64800f4850 2024-02-20T13:49:14.173 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.187+0000 7f6475ffb700 1 -- 172.21.15.192:0/4168622182 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f647001d7f0 con 0x7f64800f4850 2024-02-20T13:49:14.174 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:14.174 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 msgr2=0x7f646c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 0x7f646c033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f647c0096a0 tx=0x7f647c006b40).stop 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 msgr2=0x7f64801067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f64700089f0 tx=0x7f647000b040).stop 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 shutdown_connections 2024-02-20T13:49:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:14.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 --2- 172.21.15.192:0/4168622182 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800f4850 0x7f64801067b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:14.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 >> 172.21.15.192:0/4168622182 conn(0x7f64800f0390 msgr2=0x7f64800f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:14.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 shutdown_connections 2024-02-20T13:49:14.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:14.189+0000 7f64877bf700 1 -- 172.21.15.192:0/4168622182 wait complete. 2024-02-20T13:49:14.176 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:15.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:13.403920+0000 mgr.smithi158.gmoobf (mgr.14156) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:15.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:14.187956+0000 mon.smithi158 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.192:0/4168622182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:15.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:15.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:17.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:15.404299+0000 mgr.smithi158.gmoobf (mgr.14156) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:17.798 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 -- 172.21.15.192:0/1967289538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 msgr2=0x7f56480f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1967289538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56480f4c70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5640009960 tx=0x7f5640004d10).stop 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 -- 172.21.15.192:0/1967289538 shutdown_connections 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1967289538 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56480f4c70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 -- 172.21.15.192:0/1967289538 >> 172.21.15.192:0/1967289538 conn(0x7f56480f0390 msgr2=0x7f56480f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 -- 172.21.15.192:0/1967289538 shutdown_connections 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.811+0000 7f564d8f5700 1 -- 172.21.15.192:0/1967289538 wait complete. 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f564d8f5700 1 Processor -- start 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f564d8f5700 1 -- start start 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f564d8f5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:17.799 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f564d8f5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5648106c90 con 0x7f56480f4850 2024-02-20T13:49:17.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:17.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51288/0 (socket says 172.21.15.192:51288) 2024-02-20T13:49:17.801 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.812+0000 7f5646ffd700 1 -- 172.21.15.192:0/1932197381 learned_addr learned my addr 172.21.15.192:0/1932197381 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f5646ffd700 1 -- 172.21.15.192:0/1932197381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5640009640 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f5646ffd700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f5640009610 tx=0x7f5640004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5640018070 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f564000b7e0 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5648106e90 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564001d420 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.813+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5648107320 con 0x7f56480f4850 2024-02-20T13:49:17.802 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.814+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f5640024060 con 0x7f56480f4850 2024-02-20T13:49:17.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.814+0000 7f563f7fe700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 0x7f5634033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:17.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.814+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5648045bb0 con 0x7f56480f4850 2024-02-20T13:49:17.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.814+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f56400425a0 con 0x7f56480f4850 2024-02-20T13:49:17.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.815+0000 7f56467fc700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 0x7f5634033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:17.803 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.815+0000 7f56467fc700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 0x7f5634033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f56300096a0 tx=0x7f5630006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:17.805 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:17.819+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f56400116a0 con 0x7f56480f4850 2024-02-20T13:49:18.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.023+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56481081d0 con 0x7f56480f4850 2024-02-20T13:49:18.010 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.023+0000 7f563f7fe700 1 -- 172.21.15.192:0/1932197381 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f56400118b0 con 0x7f56480f4850 2024-02-20T13:49:18.010 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:18.010 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 msgr2=0x7f5634033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 0x7f5634033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f56300096a0 tx=0x7f5630006b40).stop 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 msgr2=0x7f56481067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f5640009610 tx=0x7f5640004d10).stop 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 shutdown_connections 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5634031090 0x7f5634033550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 --2- 172.21.15.192:0/1932197381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56480f4850 0x7f56481067a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 >> 172.21.15.192:0/1932197381 conn(0x7f56480f0390 msgr2=0x7f56480f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 shutdown_connections 2024-02-20T13:49:18.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:18.025+0000 7f564d8f5700 1 -- 172.21.15.192:0/1932197381 wait complete. 2024-02-20T13:49:18.013 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:18.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:18.024844+0000 mon.smithi158 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.192:0/1932197381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:19.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:17.404684+0000 mgr.smithi158.gmoobf (mgr.14156) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:19.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:19.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:21.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:19.405051+0000 mgr.smithi158.gmoobf (mgr.14156) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:21.678 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 -- 172.21.15.192:0/3374860465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 msgr2=0x7f81b00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:21.678 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 --2- 172.21.15.192:0/3374860465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b00f58a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f81a0009fb0 tx=0x7f81a0009b30).stop 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 -- 172.21.15.192:0/3374860465 shutdown_connections 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 --2- 172.21.15.192:0/3374860465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b00f58a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 -- 172.21.15.192:0/3374860465 >> 172.21.15.192:0/3374860465 conn(0x7f81b00f0be0 msgr2=0x7f81b00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 -- 172.21.15.192:0/3374860465 shutdown_connections 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.690+0000 7f81b7774700 1 -- 172.21.15.192:0/3374860465 wait complete. 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.691+0000 7f81b7774700 1 Processor -- start 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.691+0000 7f81b7774700 1 -- start start 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.691+0000 7f81b7774700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.691+0000 7f81b7774700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0060360 con 0x7f81b00f5480 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b5510700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b5510700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51298/0 (socket says 172.21.15.192:51298) 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b5510700 1 -- 172.21.15.192:0/3538404404 learned_addr learned my addr 172.21.15.192:0/3538404404 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b5510700 1 -- 172.21.15.192:0/3538404404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0005040 con 0x7f81b00f5480 2024-02-20T13:49:21.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b5510700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f81a0000c00 tx=0x7f81a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a00095f0 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b0060560 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f81a0007e50 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a0014e70 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.692+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b0109be0 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.693+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f81a0025070 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.694+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b0045bb0 con 0x7f81b00f5480 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.694+0000 7f81a5ffb700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 0x7f819c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.694+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f81a0041d10 con 0x7f81b00f5480 2024-02-20T13:49:21.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.694+0000 7f81b4d0f700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 0x7f819c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:21.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.694+0000 7f81b4d0f700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 0x7f819c033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f81ac0096a0 tx=0x7f81ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:21.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.699+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f81a000f4e0 con 0x7f81b00f5480 2024-02-20T13:49:21.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.904+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81b00faf70 con 0x7f81b00f5480 2024-02-20T13:49:21.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.905+0000 7f81a5ffb700 1 -- 172.21.15.192:0/3538404404 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f81b00faf70 con 0x7f81b00f5480 2024-02-20T13:49:21.892 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:21.892 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 msgr2=0x7f819c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 0x7f819c033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f81ac0096a0 tx=0x7f81ac006b40).stop 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 msgr2=0x7f81b01094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f81a0000c00 tx=0x7f81a000b040).stop 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 shutdown_connections 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f819c031090 0x7f819c033550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 --2- 172.21.15.192:0/3538404404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f81b00f5480 0x7f81b01094b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 >> 172.21.15.192:0/3538404404 conn(0x7f81b00f0be0 msgr2=0x7f81b00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 shutdown_connections 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:21.907+0000 7f81b7774700 1 -- 172.21.15.192:0/3538404404 wait complete. 2024-02-20T13:49:21.894 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:22.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:21.906563+0000 mon.smithi158 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.192:0/3538404404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:23.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:21.405388+0000 mgr.smithi158.gmoobf (mgr.14156) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:23.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:23.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:23.405713+0000 mgr.smithi158.gmoobf (mgr.14156) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:25.666 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 -- 172.21.15.192:0/3317868536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 msgr2=0x7f2b5c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 --2- 172.21.15.192:0/3317868536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c0f56f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f2b58009fb0 tx=0x7f2b58009b30).stop 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 -- 172.21.15.192:0/3317868536 shutdown_connections 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 --2- 172.21.15.192:0/3317868536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c0f56f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 -- 172.21.15.192:0/3317868536 >> 172.21.15.192:0/3317868536 conn(0x7f2b5c0f0a10 msgr2=0x7f2b5c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 -- 172.21.15.192:0/3317868536 shutdown_connections 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 -- 172.21.15.192:0/3317868536 wait complete. 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.677+0000 7f2b62c09700 1 Processor -- start 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b62c09700 1 -- start start 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b62c09700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:25.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b62c09700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c105a30 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b61c07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b61c07700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39900/0 (socket says 172.21.15.192:39900) 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b61c07700 1 -- 172.21.15.192:0/1789679922 learned_addr learned my addr 172.21.15.192:0/1789679922 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b61c07700 1 -- 172.21.15.192:0/1789679922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58005040 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b61c07700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f2b580089f0 tx=0x7f2b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.678+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b58009320 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c105c30 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2b58013ec0 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b58014d30 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c1060c0 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f2b580263a0 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.679+0000 7f2b527fc700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 0x7f2b48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.680+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2b58042420 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.680+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c045bb0 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.680+0000 7f2b61406700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 0x7f2b48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:25.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.680+0000 7f2b61406700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 0x7f2b48033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:25.671 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.685+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2b580266d0 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.876 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.890+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b5c106e10 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.877 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.890+0000 7f2b527fc700 1 -- 172.21.15.192:0/1789679922 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2b5802b020 con 0x7f2b5c0f52d0 2024-02-20T13:49:25.877 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:25.877 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:25.878 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 msgr2=0x7f2b48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 0x7f2b48033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).stop 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 msgr2=0x7f2b5c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f2b580089f0 tx=0x7f2b5800b040).stop 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 shutdown_connections 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2b48031090 0x7f2b48033550 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 --2- 172.21.15.192:0/1789679922 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b5c0f52d0 0x7f2b5c105540 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 >> 172.21.15.192:0/1789679922 conn(0x7f2b5c0f0a10 msgr2=0x7f2b5c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 shutdown_connections 2024-02-20T13:49:25.879 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:25.892+0000 7f2b62c09700 1 -- 172.21.15.192:0/1789679922 wait complete. 2024-02-20T13:49:25.880 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:26.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:25.891690+0000 mon.smithi158 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.192:0/1789679922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:25.406084+0000 mgr.smithi158.gmoobf (mgr.14156) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:27.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:27.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 -- 172.21.15.192:0/2866070612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 msgr2=0x7f48040f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 --2- 172.21.15.192:0/2866070612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f48040f6800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f47f4009fb0 tx=0x7f47f4009b30).stop 2024-02-20T13:49:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 -- 172.21.15.192:0/2866070612 shutdown_connections 2024-02-20T13:49:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 --2- 172.21.15.192:0/2866070612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f48040f6800 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 -- 172.21.15.192:0/2866070612 >> 172.21.15.192:0/2866070612 conn(0x7f48040f1b40 msgr2=0x7f48040f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 -- 172.21.15.192:0/2866070612 shutdown_connections 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 -- 172.21.15.192:0/2866070612 wait complete. 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.394+0000 7f480359e700 1 Processor -- start 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480359e700 1 -- start start 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480359e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480359e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4804106b10 con 0x7f48040f63e0 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39920/0 (socket says 172.21.15.192:39920) 2024-02-20T13:49:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480259c700 1 -- 172.21.15.192:0/1509177263 learned_addr learned my addr 172.21.15.192:0/1509177263 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480259c700 1 -- 172.21.15.192:0/1509177263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4005040 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480259c700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f47f40089f0 tx=0x7f47f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f4009320 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804106d10 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f47f4013ec0 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.395+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041071a0 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.396+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f4014d30 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.396+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f47f40095f0 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.396+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804045bb0 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.396+0000 7f47f2ffd700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 0x7f47ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.397+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47f4042420 con 0x7f48040f63e0 2024-02-20T13:49:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.397+0000 7f4801d9b700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 0x7f47ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:29.385 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.397+0000 7f4801d9b700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 0x7f47ec033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f47f80096a0 tx=0x7f47f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:29.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.401+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f47f40268c0 con 0x7f48040f63e0 2024-02-20T13:49:29.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:27.406482+0000 mgr.smithi158.gmoobf (mgr.14156) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:29.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.606+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4804107ff0 con 0x7f48040f63e0 2024-02-20T13:49:29.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.607+0000 7f47f2ffd700 1 -- 172.21.15.192:0/1509177263 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f47f401d7f0 con 0x7f48040f63e0 2024-02-20T13:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:29.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 msgr2=0x7f47ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 0x7f47ec033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f47f80096a0 tx=0x7f47f8006b40).stop 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 msgr2=0x7f4804106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f47f40089f0 tx=0x7f47f400b040).stop 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 shutdown_connections 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f47ec031090 0x7f47ec033550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 --2- 172.21.15.192:0/1509177263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040f63e0 0x7f4804106620 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 >> 172.21.15.192:0/1509177263 conn(0x7f48040f1b40 msgr2=0x7f48040f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 shutdown_connections 2024-02-20T13:49:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:29.609+0000 7f480359e700 1 -- 172.21.15.192:0/1509177263 wait complete. 2024-02-20T13:49:29.597 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:30.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:29.608466+0000 mon.smithi158 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.192:0/1509177263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:31.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:31.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:31.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:29.406815+0000 mgr.smithi158.gmoobf (mgr.14156) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.200+0000 7f57ad6c0700 1 -- 172.21.15.192:0/806167042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 msgr2=0x7f57a80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.200+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/806167042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a80f56f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5798009fb0 tx=0x7f5798009b30).stop 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.200+0000 7f57ad6c0700 1 -- 172.21.15.192:0/806167042 shutdown_connections 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.200+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/806167042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a80f56f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.200+0000 7f57ad6c0700 1 -- 172.21.15.192:0/806167042 >> 172.21.15.192:0/806167042 conn(0x7f57a80f0a30 msgr2=0x7f57a80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 -- 172.21.15.192:0/806167042 shutdown_connections 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 -- 172.21.15.192:0/806167042 wait complete. 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 Processor -- start 2024-02-20T13:49:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 -- start start 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.201+0000 7f57ad6c0700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8106af0 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a7fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56208/0 (socket says 172.21.15.192:56208) 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a7fff700 1 -- 172.21.15.192:0/2677076574 learned_addr learned my addr 172.21.15.192:0/2677076574 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a7fff700 1 -- 172.21.15.192:0/2677076574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798005040 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a7fff700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5798000c00 tx=0x7f579800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57980095f0 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5798007e50 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5798014e70 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a8106cf0 con 0x7f57a80f52d0 2024-02-20T13:49:33.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.202+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a8107180 con 0x7f57a80f52d0 2024-02-20T13:49:33.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.203+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f57980079c0 con 0x7f57a80f52d0 2024-02-20T13:49:33.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.203+0000 7f57a4ff9700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 0x7f5790033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:33.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.203+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a8045bb0 con 0x7f57a80f52d0 2024-02-20T13:49:33.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.203+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5798041cf0 con 0x7f57a80f52d0 2024-02-20T13:49:33.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.204+0000 7f57a77fe700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 0x7f5790033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:33.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.204+0000 7f57a77fe700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 0x7f5790033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f579c0096a0 tx=0x7f579c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:33.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.208+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f579801b730 con 0x7f57a80f52d0 2024-02-20T13:49:33.402 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.416+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57a8107f80 con 0x7f57a80f52d0 2024-02-20T13:49:33.403 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.417+0000 7f57a4ff9700 1 -- 172.21.15.192:0/2677076574 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f579801b730 con 0x7f57a80f52d0 2024-02-20T13:49:33.403 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:33.404 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 msgr2=0x7f5790033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 0x7f5790033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f579c0096a0 tx=0x7f579c006b40).stop 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 msgr2=0x7f57a8106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5798000c00 tx=0x7f579800b040).stop 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 shutdown_connections 2024-02-20T13:49:33.405 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5790031090 0x7f5790033550 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:33.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 --2- 172.21.15.192:0/2677076574 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57a80f52d0 0x7f57a8106600 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:33.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 >> 172.21.15.192:0/2677076574 conn(0x7f57a80f0a30 msgr2=0x7f57a805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:33.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 shutdown_connections 2024-02-20T13:49:33.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:33.418+0000 7f57ad6c0700 1 -- 172.21.15.192:0/2677076574 wait complete. 2024-02-20T13:49:33.406 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:33.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:31.407113+0000 mgr.smithi158.gmoobf (mgr.14156) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:34.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:33.417990+0000 mon.smithi158 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.192:0/2677076574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:35.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:35.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:35.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:33.407447+0000 mgr.smithi158.gmoobf (mgr.14156) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 -- 172.21.15.192:0/577564612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 msgr2=0x7f775c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 --2- 172.21.15.192:0/577564612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c0f8f80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f7750009fb0 tx=0x7f7750009b30).stop 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 -- 172.21.15.192:0/577564612 shutdown_connections 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 --2- 172.21.15.192:0/577564612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c0f8f80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 -- 172.21.15.192:0/577564612 >> 172.21.15.192:0/577564612 conn(0x7f775c0f09a0 msgr2=0x7f775c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 -- 172.21.15.192:0/577564612 shutdown_connections 2024-02-20T13:49:36.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.962+0000 7f7761ade700 1 -- 172.21.15.192:0/577564612 wait complete. 2024-02-20T13:49:36.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7761ade700 1 Processor -- start 2024-02-20T13:49:36.950 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7761ade700 1 -- start start 2024-02-20T13:49:36.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7761ade700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:36.951 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7761ade700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c106a30 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7760adc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7760adc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56230/0 (socket says 172.21.15.192:56230) 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7760adc700 1 -- 172.21.15.192:0/2491585369 learned_addr learned my addr 172.21.15.192:0/2491585369 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.963+0000 7f7760adc700 1 -- 172.21.15.192:0/2491585369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750005040 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f7760adc700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f7750009f80 tx=0x7f775000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7750009320 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c106c30 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7750013680 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775001eda0 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.964+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c1070c0 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f775001e420 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f77597fa700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 0x7f774c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7750042340 con 0x7f775c060300 2024-02-20T13:49:36.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c045bb0 con 0x7f775c060300 2024-02-20T13:49:36.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f775bfff700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 0x7f774c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:36.953 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.965+0000 7f775bfff700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 0x7f774c033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f77480096a0 tx=0x7f7748006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:36.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:36.970+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f775001e700 con 0x7f775c060300 2024-02-20T13:49:37.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.177+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f775c0fbe20 con 0x7f775c060300 2024-02-20T13:49:37.164 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.178+0000 7f77597fa700 1 -- 172.21.15.192:0/2491585369 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f775001cb10 con 0x7f775c060300 2024-02-20T13:49:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:37.165 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:37.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 msgr2=0x7f774c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:37.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 0x7f774c033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f77480096a0 tx=0x7f7748006b40).stop 2024-02-20T13:49:37.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 msgr2=0x7f775c106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:37.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f7750009f80 tx=0x7f775000b040).stop 2024-02-20T13:49:37.166 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 shutdown_connections 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f774c031090 0x7f774c033550 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 --2- 172.21.15.192:0/2491585369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f775c060300 0x7f775c106540 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 >> 172.21.15.192:0/2491585369 conn(0x7f775c0f09a0 msgr2=0x7f775c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 shutdown_connections 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:37.180+0000 7f7761ade700 1 -- 172.21.15.192:0/2491585369 wait complete. 2024-02-20T13:49:37.167 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:37.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:35.407822+0000 mgr.smithi158.gmoobf (mgr.14156) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:37.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:37.179579+0000 mon.smithi158 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.192:0/2491585369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:38.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:38.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:39.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:37.408130+0000 mgr.smithi158.gmoobf (mgr.14156) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 -- 172.21.15.192:0/3304397554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 msgr2=0x7fcd900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 --2- 172.21.15.192:0/3304397554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd900f58a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcd80009fb0 tx=0x7fcd80009b30).stop 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 -- 172.21.15.192:0/3304397554 shutdown_connections 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 --2- 172.21.15.192:0/3304397554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd900f58a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 -- 172.21.15.192:0/3304397554 >> 172.21.15.192:0/3304397554 conn(0x7fcd900f0be0 msgr2=0x7fcd900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 -- 172.21.15.192:0/3304397554 shutdown_connections 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.835+0000 7fcd963a2700 1 -- 172.21.15.192:0/3304397554 wait complete. 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.836+0000 7fcd963a2700 1 Processor -- start 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.836+0000 7fcd963a2700 1 -- start start 2024-02-20T13:49:40.824 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.836+0000 7fcd963a2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.836+0000 7fcd963a2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90106ce0 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:56252/0 (socket says 172.21.15.192:56252) 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8ffff700 1 -- 172.21.15.192:0/2494114250 learned_addr learned my addr 172.21.15.192:0/2494114250 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8ffff700 1 -- 172.21.15.192:0/2494114250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80005040 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8ffff700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fcd80000c00 tx=0x7fcd8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd800095f0 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcd80007e50 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd90106ee0 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd80014e70 con 0x7fcd900f5480 2024-02-20T13:49:40.825 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.837+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd90107370 con 0x7fcd900f5480 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.838+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fcd80025070 con 0x7fcd900f5480 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.838+0000 7fcd8cff9700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 0x7fcd7c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.838+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcd80041d70 con 0x7fcd900f5480 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.838+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd90045bb0 con 0x7fcd900f5480 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.838+0000 7fcd8f7fe700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 0x7fcd7c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:40.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.839+0000 7fcd8f7fe700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 0x7fcd7c033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcd780096a0 tx=0x7fcd78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:40.829 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:40.843+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcd800079c0 con 0x7fcd900f5480 2024-02-20T13:49:41.038 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.051+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd900fb060 con 0x7fcd900f5480 2024-02-20T13:49:41.038 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.052+0000 7fcd8cff9700 1 -- 172.21.15.192:0/2494114250 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd80025300 con 0x7fcd900f5480 2024-02-20T13:49:41.039 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:41.039 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 msgr2=0x7fcd7c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 0x7fcd7c033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcd780096a0 tx=0x7fcd78006b40).stop 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 msgr2=0x7fcd901067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fcd80000c00 tx=0x7fcd8000b040).stop 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 shutdown_connections 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fcd7c031090 0x7fcd7c033550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:41.041 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 --2- 172.21.15.192:0/2494114250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd900f5480 0x7fcd901067f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:41.042 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 >> 172.21.15.192:0/2494114250 conn(0x7fcd900f0be0 msgr2=0x7fcd900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:41.042 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 shutdown_connections 2024-02-20T13:49:41.042 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:41.054+0000 7fcd963a2700 1 -- 172.21.15.192:0/2494114250 wait complete. 2024-02-20T13:49:41.042 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:41.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:39.408472+0000 mgr.smithi158.gmoobf (mgr.14156) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:41.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:41.053309+0000 mon.smithi158 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.192:0/2494114250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:42.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:42.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:43.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:41.408808+0000 mgr.smithi158.gmoobf (mgr.14156) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/1726529203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 msgr2=0x7fd6e80f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/1726529203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80f5e80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fd6dc009fb0 tx=0x7fd6dc009b30).stop 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/1726529203 shutdown_connections 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/1726529203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80f5e80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/1726529203 >> 172.21.15.192:0/1726529203 conn(0x7fd6e80f0390 msgr2=0x7fd6e80f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/1726529203 shutdown_connections 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.685+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/1726529203 wait complete. 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6ed4d4700 1 Processor -- start 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6ed4d4700 1 -- start start 2024-02-20T13:49:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6ed4d4700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6ed4d4700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80fb8f0 con 0x7fd6e80f5a60 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6e6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:46928/0 (socket says 172.21.15.192:46928) 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6e6ffd700 1 -- 172.21.15.192:0/2530416492 learned_addr learned my addr 172.21.15.192:0/2530416492 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6e6ffd700 1 -- 172.21.15.192:0/2530416492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc005040 con 0x7fd6e80f5a60 2024-02-20T13:49:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.686+0000 7fd6e6ffd700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fd6dc000c00 tx=0x7fd6dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:44.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.687+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6dc0095f0 con 0x7fd6e80f5a60 2024-02-20T13:49:44.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.687+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e80fba90 con 0x7fd6e80f5a60 2024-02-20T13:49:44.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.687+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e80fbf20 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.687+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd6dc007e50 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.687+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6dc014e00 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fd6dc025070 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6cf7fe700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 0x7fd6d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd6dc042020 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e80591e0 con 0x7fd6e80f5a60 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6e67fc700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 0x7fd6d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.688+0000 7fd6e67fc700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 0x7fd6d0033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd6d80096a0 tx=0x7fd6d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:44.679 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.692+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd6dc00f300 con 0x7fd6e80f5a60 2024-02-20T13:49:44.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.901+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6e8045bb0 con 0x7fd6e80f5a60 2024-02-20T13:49:44.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.902+0000 7fd6cf7fe700 1 -- 172.21.15.192:0/2530416492 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd6dc025af0 con 0x7fd6e80f5a60 2024-02-20T13:49:44.888 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:44.889 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:44.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 msgr2=0x7fd6d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 0x7fd6d0033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd6d80096a0 tx=0x7fd6d8006b40).stop 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 msgr2=0x7fd6e80fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fd6dc000c00 tx=0x7fd6dc00b040).stop 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 shutdown_connections 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd6d0031090 0x7fd6d0033550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.903+0000 7fd6ed4d4700 1 --2- 172.21.15.192:0/2530416492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6e80f5a60 0x7fd6e80fd470 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.904+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 >> 172.21.15.192:0/2530416492 conn(0x7fd6e80f0390 msgr2=0x7fd6e80f5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.904+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 shutdown_connections 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:44.904+0000 7fd6ed4d4700 1 -- 172.21.15.192:0/2530416492 wait complete. 2024-02-20T13:49:44.891 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:45.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:43.409110+0000 mgr.smithi158.gmoobf (mgr.14156) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:45.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:44.903286+0000 mon.smithi158 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.192:0/2530416492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:46.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:46.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:47.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:45.409484+0000 mgr.smithi158.gmoobf (mgr.14156) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:48.498 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 -- 172.21.15.192:0/3237478643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 msgr2=0x7fc7940f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:48.498 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/3237478643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc7940f69b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc790009fb0 tx=0x7fc790009b30).stop 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 -- 172.21.15.192:0/3237478643 shutdown_connections 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/3237478643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc7940f69b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 -- 172.21.15.192:0/3237478643 >> 172.21.15.192:0/3237478643 conn(0x7fc7940f1d10 msgr2=0x7fc7940f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 -- 172.21.15.192:0/3237478643 shutdown_connections 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 -- 172.21.15.192:0/3237478643 wait complete. 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.510+0000 7fc79c6ec700 1 Processor -- start 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79c6ec700 1 -- start start 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79c6ec700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79c6ec700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794106cf0 con 0x7fc7940f6590 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79a488700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79a488700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:46940/0 (socket says 172.21.15.192:46940) 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79a488700 1 -- 172.21.15.192:0/163831722 learned_addr learned my addr 172.21.15.192:0/163831722 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79a488700 1 -- 172.21.15.192:0/163831722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790005040 con 0x7fc7940f6590 2024-02-20T13:49:48.499 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79a488700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fc7900089f0 tx=0x7fc79000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc790009320 con 0x7fc7940f6590 2024-02-20T13:49:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc794106ef0 con 0x7fc7940f6590 2024-02-20T13:49:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc790013ec0 con 0x7fc7940f6590 2024-02-20T13:49:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc790014d30 con 0x7fc7940f6590 2024-02-20T13:49:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.511+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc794107380 con 0x7fc7940f6590 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.512+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc7900095f0 con 0x7fc7940f6590 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.512+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc794045bb0 con 0x7fc7940f6590 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.512+0000 7fc78affd700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 0x7fc780033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.512+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc790042390 con 0x7fc7940f6590 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.513+0000 7fc799c87700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 0x7fc780033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:48.501 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.513+0000 7fc799c87700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 0x7fc780033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc7840096a0 tx=0x7fc784006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:48.503 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.517+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc79001d7f0 con 0x7fc7940f6590 2024-02-20T13:49:48.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.723+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7940fc080 con 0x7fc7940f6590 2024-02-20T13:49:48.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.724+0000 7fc78affd700 1 -- 172.21.15.192:0/163831722 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc79001d7f0 con 0x7fc7940f6590 2024-02-20T13:49:48.711 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:48.711 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 msgr2=0x7fc780033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 0x7fc780033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc7840096a0 tx=0x7fc784006b40).stop 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 msgr2=0x7fc794106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fc7900089f0 tx=0x7fc79000b040).stop 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 shutdown_connections 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc780031090 0x7fc780033550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:48.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 --2- 172.21.15.192:0/163831722 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7940f6590 0x7fc794106800 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:48.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 >> 172.21.15.192:0/163831722 conn(0x7fc7940f1d10 msgr2=0x7fc7940f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:48.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 shutdown_connections 2024-02-20T13:49:48.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:48.726+0000 7fc79c6ec700 1 -- 172.21.15.192:0/163831722 wait complete. 2024-02-20T13:49:48.714 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:49.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:47.409898+0000 mgr.smithi158.gmoobf (mgr.14156) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:49.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:49 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:48.725653+0000 mon.smithi158 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.192:0/163831722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:50.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:50.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:51.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:49.410288+0000 mgr.smithi158.gmoobf (mgr.14156) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:52.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/281731348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 msgr2=0x7f1a640f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/281731348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a640f58a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1a58009fb0 tx=0x7f1a58009b30).stop 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/281731348 shutdown_connections 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/281731348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a640f58a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/281731348 >> 172.21.15.192:0/281731348 conn(0x7f1a640f0be0 msgr2=0x7f1a640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/281731348 shutdown_connections 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/281731348 wait complete. 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.443+0000 7f1a6a6bc700 1 Processor -- start 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a6a6bc700 1 -- start start 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a6a6bc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a6a6bc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64060360 con 0x7f1a640f5480 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:52.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53330/0 (socket says 172.21.15.192:53330) 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a63fff700 1 -- 172.21.15.192:0/2625031850 learned_addr learned my addr 172.21.15.192:0/2625031850 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a63fff700 1 -- 172.21.15.192:0/2625031850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58005040 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.444+0000 7f1a63fff700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f1a580089f0 tx=0x7f1a5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a58009320 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a64060560 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1a58013ec0 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a58014d30 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a64109be0 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1a580095f0 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.445+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a64045bb0 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.446+0000 7f1a60ff9700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 0x7f1a4c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.446+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1a58042450 con 0x7f1a640f5480 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.446+0000 7f1a637fe700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 0x7f1a4c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:52.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.446+0000 7f1a637fe700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 0x7f1a4c033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f1a540096a0 tx=0x7f1a54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:52.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.450+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1a5801d7f0 con 0x7f1a640f5480 2024-02-20T13:49:52.641 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.655+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a6410a400 con 0x7f1a640f5480 2024-02-20T13:49:52.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.656+0000 7f1a60ff9700 1 -- 172.21.15.192:0/2625031850 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1a5801d7f0 con 0x7f1a640f5480 2024-02-20T13:49:52.642 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:52.642 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 msgr2=0x7f1a4c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 0x7f1a4c033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f1a540096a0 tx=0x7f1a54006b40).stop 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 msgr2=0x7f1a641094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f1a580089f0 tx=0x7f1a5800b040).stop 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 shutdown_connections 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1a4c031090 0x7f1a4c033550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 --2- 172.21.15.192:0/2625031850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a640f5480 0x7f1a641094b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 >> 172.21.15.192:0/2625031850 conn(0x7f1a640f0be0 msgr2=0x7f1a640f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 shutdown_connections 2024-02-20T13:49:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:52.657+0000 7f1a6a6bc700 1 -- 172.21.15.192:0/2625031850 wait complete. 2024-02-20T13:49:52.645 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:53.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:51.410595+0000 mgr.smithi158.gmoobf (mgr.14156) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:53.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:52.657196+0000 mon.smithi158 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.192:0/2625031850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:54.434 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:54.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:55.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:53.410909+0000 mgr.smithi158.gmoobf (mgr.14156) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- 172.21.15.192:0/517250310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 msgr2=0x7f308c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 --2- 172.21.15.192:0/517250310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c0f58a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f307c009fb0 tx=0x7f307c009b30).stop 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- 172.21.15.192:0/517250310 shutdown_connections 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 --2- 172.21.15.192:0/517250310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c0f58a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- 172.21.15.192:0/517250310 >> 172.21.15.192:0/517250310 conn(0x7f308c0f0be0 msgr2=0x7f308c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- 172.21.15.192:0/517250310 shutdown_connections 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- 172.21.15.192:0/517250310 wait complete. 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 Processor -- start 2024-02-20T13:49:56.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- start start 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.408+0000 7f30934bc700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c060300 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3091258700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3091258700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53342/0 (socket says 172.21.15.192:53342) 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3091258700 1 -- 172.21.15.192:0/735691227 learned_addr learned my addr 172.21.15.192:0/735691227 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3091258700 1 -- 172.21.15.192:0/735691227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c005040 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3091258700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f307c0089f0 tx=0x7f307c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307c009320 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f307c013ec0 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c060500 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307c014d30 con 0x7f308c0f5480 2024-02-20T13:49:56.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.409+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c109bc0 con 0x7f308c0f5480 2024-02-20T13:49:56.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.410+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f307c0095f0 con 0x7f308c0f5480 2024-02-20T13:49:56.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.410+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c0591e0 con 0x7f308c0f5480 2024-02-20T13:49:56.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.410+0000 7f3081ffb700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 0x7f3078033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:49:56.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.411+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f307c042450 con 0x7f308c0f5480 2024-02-20T13:49:56.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.411+0000 7f3090a57700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 0x7f3078033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:49:56.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.411+0000 7f3090a57700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 0x7f3078033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f30880096a0 tx=0x7f3088006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:49:56.401 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.416+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f307c01d7f0 con 0x7f308c0f5480 2024-02-20T13:49:56.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:55.467711+0000 mon.smithi158 (mon.0) 272 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:49:56.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:55.467936+0000 mon.smithi158 (mon.0) 273 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:49:56.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.620+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f308c045bb0 con 0x7f308c0f5480 2024-02-20T13:49:56.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.621+0000 7f3081ffb700 1 -- 172.21.15.192:0/735691227 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f307c01d7f0 con 0x7f308c0f5480 2024-02-20T13:49:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:49:56.607 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.622+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 msgr2=0x7f3078033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.622+0000 7f30934bc700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 0x7f3078033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f30880096a0 tx=0x7f3088006b40).stop 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.622+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 msgr2=0x7f308c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.622+0000 7f30934bc700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f307c0089f0 tx=0x7f307c00b040).stop 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 shutdown_connections 2024-02-20T13:49:56.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3078031090 0x7f3078033550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:56.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 --2- 172.21.15.192:0/735691227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f308c0f5480 0x7f308c109510 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:49:56.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 >> 172.21.15.192:0/735691227 conn(0x7f308c0f0be0 msgr2=0x7f308c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:49:56.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 shutdown_connections 2024-02-20T13:49:56.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:49:56.623+0000 7f30934bc700 1 -- 172.21.15.192:0/735691227 wait complete. 2024-02-20T13:49:56.609 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:49:57.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:55.411257+0000 mgr.smithi158.gmoobf (mgr.14156) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:49:57.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:49:56.621364+0000 mon.smithi158 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.192:0/735691227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:49:58.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:49:58.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:49:59.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:49:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:57.411564+0000 mgr.smithi158.gmoobf (mgr.14156) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- 172.21.15.192:0/2693113905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 msgr2=0x7fa0380f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 --2- 172.21.15.192:0/2693113905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa0380f69b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fa02c004760 tx=0x7fa02c009ad0).stop 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- 172.21.15.192:0/2693113905 shutdown_connections 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 --2- 172.21.15.192:0/2693113905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa0380f69b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- 172.21.15.192:0/2693113905 >> 172.21.15.192:0/2693113905 conn(0x7fa0380f1cd0 msgr2=0x7fa0380f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- 172.21.15.192:0/2693113905 shutdown_connections 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- 172.21.15.192:0/2693113905 wait complete. 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 Processor -- start 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- start start 2024-02-20T13:50:00.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.082+0000 7fa03f65a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038069f00 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03d3f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03d3f6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53360/0 (socket says 172.21.15.192:53360) 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03d3f6700 1 -- 172.21.15.192:0/295525253 learned_addr learned my addr 172.21.15.192:0/295525253 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03d3f6700 1 -- 172.21.15.192:0/295525253 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c009d70 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03d3f6700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fa02c004170 tx=0x7fa02c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c00e040 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03806a0a0 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa02c008ef0 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c014c90 con 0x7fa0380f6590 2024-02-20T13:50:00.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.083+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03806a530 con 0x7fa0380f6590 2024-02-20T13:50:00.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.084+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa02c014420 con 0x7fa0380f6590 2024-02-20T13:50:00.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.084+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0380591e0 con 0x7fa0380f6590 2024-02-20T13:50:00.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.085+0000 7fa029ffb700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 0x7fa0240335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:00.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.085+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa02c0422e0 con 0x7fa0380f6590 2024-02-20T13:50:00.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.085+0000 7fa03cbf5700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 0x7fa0240335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:00.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.085+0000 7fa03cbf5700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 0x7fa0240335a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa0340096a0 tx=0x7fa034006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:00.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.089+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa02c026300 con 0x7fa0380f6590 2024-02-20T13:50:00.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.296+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0380fc080 con 0x7fa0380f6590 2024-02-20T13:50:00.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.297+0000 7fa029ffb700 1 -- 172.21.15.192:0/295525253 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa02c012b20 con 0x7fa0380f6590 2024-02-20T13:50:00.282 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:00.282 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 msgr2=0x7fa0240335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 0x7fa0240335a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa0340096a0 tx=0x7fa034006b40).stop 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 msgr2=0x7fa03806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fa02c004170 tx=0x7fa02c009ad0).stop 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 shutdown_connections 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa0240310e0 0x7fa0240335a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:00.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 --2- 172.21.15.192:0/295525253 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0380f6590 0x7fa03806ba80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:00.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 >> 172.21.15.192:0/295525253 conn(0x7fa0380f1cd0 msgr2=0x7fa0380f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:00.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 shutdown_connections 2024-02-20T13:50:00.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:00.298+0000 7fa03f65a700 1 -- 172.21.15.192:0/295525253 wait complete. 2024-02-20T13:50:00.288 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:01.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:49:59.411961+0000 mgr.smithi158.gmoobf (mgr.14156) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:01.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:00.296996+0000 mon.smithi158 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.192:0/295525253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:02.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:02.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:03.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:01.412272+0000 mgr.smithi158.gmoobf (mgr.14156) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:04.017 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 -- 172.21.15.192:0/1468485821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 msgr2=0x7fde680f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/1468485821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde680f4c60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fde58009fb0 tx=0x7fde58009b30).stop 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 -- 172.21.15.192:0/1468485821 shutdown_connections 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/1468485821 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde680f4c60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 -- 172.21.15.192:0/1468485821 >> 172.21.15.192:0/1468485821 conn(0x7fde680f0360 msgr2=0x7fde680f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 -- 172.21.15.192:0/1468485821 shutdown_connections 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.015+0000 7fde6ce5b700 1 -- 172.21.15.192:0/1468485821 wait complete. 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6ce5b700 1 Processor -- start 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6ce5b700 1 -- start start 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6ce5b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6ce5b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68106cb0 con 0x7fde680f4840 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:04.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51540/0 (socket says 172.21.15.192:51540) 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6659c700 1 -- 172.21.15.192:0/723848840 learned_addr learned my addr 172.21.15.192:0/723848840 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.016+0000 7fde6659c700 1 -- 172.21.15.192:0/723848840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58005040 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde6659c700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fde580089f0 tx=0x7fde5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde58009320 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68106eb0 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fde58013ec0 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68107340 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.017+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde58014d30 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fde580095f0 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68045bb0 con 0x7fde680f4840 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde56ffd700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:04.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fde58042420 con 0x7fde680f4840 2024-02-20T13:50:04.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde65d9b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:04.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.018+0000 7fde65d9b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 0x7fde50033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:04.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.023+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde5801d7f0 con 0x7fde680f4840 2024-02-20T13:50:04.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.231+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde68108190 con 0x7fde680f4840 2024-02-20T13:50:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.232+0000 7fde56ffd700 1 -- 172.21.15.192:0/723848840 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde5801d7f0 con 0x7fde680f4840 2024-02-20T13:50:04.217 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:04.217 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 msgr2=0x7fde50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 0x7fde50033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).stop 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 msgr2=0x7fde681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fde580089f0 tx=0x7fde5800b040).stop 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 shutdown_connections 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 --2- 172.21.15.192:0/723848840 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde680f4840 0x7fde681067c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 >> 172.21.15.192:0/723848840 conn(0x7fde680f0360 msgr2=0x7fde680f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 shutdown_connections 2024-02-20T13:50:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:04.233+0000 7fde6ce5b700 1 -- 172.21.15.192:0/723848840 wait complete. 2024-02-20T13:50:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:05.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:03.412621+0000 mgr.smithi158.gmoobf (mgr.14156) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:05.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:04.231973+0000 mon.smithi158 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.192:0/723848840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:05.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:05.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:07.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:05.412933+0000 mgr.smithi158.gmoobf (mgr.14156) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:07.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 -- 172.21.15.192:0/1180561339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 msgr2=0x7f9cac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:07.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 --2- 172.21.15.192:0/1180561339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac0f69b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f9c9c004760 tx=0x7f9c9c009ad0).stop 2024-02-20T13:50:07.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 -- 172.21.15.192:0/1180561339 shutdown_connections 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 --2- 172.21.15.192:0/1180561339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac0f69b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 -- 172.21.15.192:0/1180561339 >> 172.21.15.192:0/1180561339 conn(0x7f9cac0f1cd0 msgr2=0x7f9cac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 -- 172.21.15.192:0/1180561339 shutdown_connections 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.903+0000 7f9cb3676700 1 -- 172.21.15.192:0/1180561339 wait complete. 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb3676700 1 Processor -- start 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb3676700 1 -- start start 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb3676700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:07.891 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb3676700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac106cb0 con 0x7f9cac0f6590 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb1412700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb1412700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51564/0 (socket says 172.21.15.192:51564) 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.904+0000 7f9cb1412700 1 -- 172.21.15.192:0/3563274262 learned_addr learned my addr 172.21.15.192:0/3563274262 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9cb1412700 1 -- 172.21.15.192:0/3563274262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c009d70 con 0x7f9cac0f6590 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9cb1412700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f9c9c004170 tx=0x7f9c9c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c00e040 con 0x7f9cac0f6590 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac106eb0 con 0x7f9cac0f6590 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9c9c008ef0 con 0x7f9cac0f6590 2024-02-20T13:50:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c014c90 con 0x7f9cac0f6590 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.905+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac107340 con 0x7f9cac0f6590 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.906+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9c9c014420 con 0x7f9cac0f6590 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.906+0000 7f9ca1ffb700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 0x7f9c980335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.906+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9c9c0422e0 con 0x7f9cac0f6590 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.906+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac045bb0 con 0x7f9cac0f6590 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.906+0000 7f9cb0c11700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 0x7f9c980335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:07.894 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.907+0000 7f9cb0c11700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 0x7f9c980335a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9ca80096a0 tx=0x7f9ca8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:07.897 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:07.911+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9c9c026300 con 0x7f9cac0f6590 2024-02-20T13:50:08.105 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.118+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cac0fc170 con 0x7f9cac0f6590 2024-02-20T13:50:08.105 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.119+0000 7f9ca1ffb700 1 -- 172.21.15.192:0/3563274262 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c9c012b20 con 0x7f9cac0f6590 2024-02-20T13:50:08.105 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:08.106 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 msgr2=0x7f9c980335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 0x7f9c980335a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9ca80096a0 tx=0x7f9ca8006b40).stop 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 msgr2=0x7f9cac1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f9c9c004170 tx=0x7f9c9c009ad0).stop 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 shutdown_connections 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9c980310e0 0x7f9c980335a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 --2- 172.21.15.192:0/3563274262 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cac0f6590 0x7f9cac1067c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 >> 172.21.15.192:0/3563274262 conn(0x7f9cac0f1cd0 msgr2=0x7f9cac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.120+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 shutdown_connections 2024-02-20T13:50:08.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:08.121+0000 7f9cb3676700 1 -- 172.21.15.192:0/3563274262 wait complete. 2024-02-20T13:50:08.108 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:08.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:08 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:08.120347+0000 mon.smithi158 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.192:0/3563274262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:09.469 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:07.413276+0000 mgr.smithi158.gmoobf (mgr.14156) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:09.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:09.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:11.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:09.413589+0000 mgr.smithi158.gmoobf (mgr.14156) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- 172.21.15.192:0/2692211969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 msgr2=0x7f9a800f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/2692211969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a800f56f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f9a70009fb0 tx=0x7f9a70009b30).stop 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- 172.21.15.192:0/2692211969 shutdown_connections 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/2692211969 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a800f56f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- 172.21.15.192:0/2692211969 >> 172.21.15.192:0/2692211969 conn(0x7f9a800f0a30 msgr2=0x7f9a800f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- 172.21.15.192:0/2692211969 shutdown_connections 2024-02-20T13:50:11.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- 172.21.15.192:0/2692211969 wait complete. 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 Processor -- start 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 -- start start 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.555+0000 7f9a7ffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7ffff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80106b10 con 0x7f9a800f52d0 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7effd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51582/0 (socket says 172.21.15.192:51582) 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7effd700 1 -- 172.21.15.192:0/3645389116 learned_addr learned my addr 172.21.15.192:0/3645389116 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7effd700 1 -- 172.21.15.192:0/3645389116 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70005040 con 0x7f9a800f52d0 2024-02-20T13:50:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7effd700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9a700089f0 tx=0x7f9a7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:11.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a70009320 con 0x7f9a800f52d0 2024-02-20T13:50:11.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a80106d10 con 0x7f9a800f52d0 2024-02-20T13:50:11.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a70013ec0 con 0x7f9a800f52d0 2024-02-20T13:50:11.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a70014d30 con 0x7f9a800f52d0 2024-02-20T13:50:11.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.556+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a801071a0 con 0x7f9a800f52d0 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.557+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9a700095f0 con 0x7f9a800f52d0 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.557+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a80045bb0 con 0x7f9a800f52d0 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.557+0000 7f9a677fe700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 0x7f9a68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.557+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a700424e0 con 0x7f9a800f52d0 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.558+0000 7f9a7e7fc700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 0x7f9a68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.558+0000 7f9a7e7fc700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 0x7f9a68033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:11.548 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.562+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a7001d7f0 con 0x7f9a800f52d0 2024-02-20T13:50:11.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.769+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a80107ff0 con 0x7f9a800f52d0 2024-02-20T13:50:11.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.770+0000 7f9a677fe700 1 -- 172.21.15.192:0/3645389116 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9a7001d7f0 con 0x7f9a800f52d0 2024-02-20T13:50:11.756 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:11.756 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:11.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 msgr2=0x7f9a68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:11.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 0x7f9a68033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).stop 2024-02-20T13:50:11.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 msgr2=0x7f9a80106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:11.758 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9a700089f0 tx=0x7f9a7000b040).stop 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 shutdown_connections 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9a68031090 0x7f9a68033550 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 --2- 172.21.15.192:0/3645389116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a800f52d0 0x7f9a80106620 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 >> 172.21.15.192:0/3645389116 conn(0x7f9a800f0a30 msgr2=0x7f9a8005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.771+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 shutdown_connections 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:11.772+0000 7f9a7ffff700 1 -- 172.21.15.192:0/3645389116 wait complete. 2024-02-20T13:50:11.759 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:12.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:11.771407+0000 mon.smithi158 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.192:0/3645389116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:13.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:13.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:13.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:11.413981+0000 mgr.smithi158.gmoobf (mgr.14156) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:15.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:13.414246+0000 mgr.smithi158.gmoobf (mgr.14156) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:15.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 -- 172.21.15.192:0/209480056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 msgr2=0x7fa0080f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:15.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 --2- 172.21.15.192:0/209480056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa0080f9160 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9ff8009fb0 tx=0x7f9ff8009b30).stop 2024-02-20T13:50:15.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 -- 172.21.15.192:0/209480056 shutdown_connections 2024-02-20T13:50:15.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 --2- 172.21.15.192:0/209480056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa0080f9160 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:15.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 -- 172.21.15.192:0/209480056 >> 172.21.15.192:0/209480056 conn(0x7fa0080f0be0 msgr2=0x7fa0080f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 -- 172.21.15.192:0/209480056 shutdown_connections 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 -- 172.21.15.192:0/209480056 wait complete. 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.520+0000 7fa00ea47700 1 Processor -- start 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa00ea47700 1 -- start start 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa00ea47700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa00ea47700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008106c40 con 0x7fa008060360 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa007fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa007fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40782/0 (socket says 172.21.15.192:40782) 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa007fff700 1 -- 172.21.15.192:0/1971553954 learned_addr learned my addr 172.21.15.192:0/1971553954 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.521+0000 7fa007fff700 1 -- 172.21.15.192:0/1971553954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff8005040 con 0x7fa008060360 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa007fff700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9ff8009f80 tx=0x7f9ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff8009320 con 0x7fa008060360 2024-02-20T13:50:15.540 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa008106e40 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9ff8013680 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff801eda0 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.522+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0081072d0 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9ff801e420 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa004ff9700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 0x7f9ff0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa008045bb0 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9ff8042300 con 0x7fa008060360 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa0077fe700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 0x7f9ff0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.523+0000 7fa0077fe700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 0x7f9ff0033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9ffc0096a0 tx=0x7f9ffc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:15.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.528+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9ff801ccf0 con 0x7fa008060360 2024-02-20T13:50:15.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.733+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa008108150 con 0x7fa008060360 2024-02-20T13:50:15.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.734+0000 7fa004ff9700 1 -- 172.21.15.192:0/1971553954 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ff801cb10 con 0x7fa008060360 2024-02-20T13:50:15.720 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:15.720 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:15.721 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 msgr2=0x7f9ff0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 0x7f9ff0033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9ffc0096a0 tx=0x7f9ffc006b40).stop 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 msgr2=0x7fa008106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9ff8009f80 tx=0x7f9ff800b040).stop 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 shutdown_connections 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9ff0031090 0x7f9ff0033550 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 --2- 172.21.15.192:0/1971553954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa008060360 0x7fa008106750 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 >> 172.21.15.192:0/1971553954 conn(0x7fa0080f0be0 msgr2=0x7fa0080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 shutdown_connections 2024-02-20T13:50:15.722 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:15.735+0000 7fa00ea47700 1 -- 172.21.15.192:0/1971553954 wait complete. 2024-02-20T13:50:15.723 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:16.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:15.735099+0000 mon.smithi158 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.192:0/1971553954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:17.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:17.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:17.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:15.414606+0000 mgr.smithi158.gmoobf (mgr.14156) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 -- 172.21.15.192:0/4085817946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 msgr2=0x7f92440f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 --2- 172.21.15.192:0/4085817946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440f58a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f9238009fb0 tx=0x7f9238009b30).stop 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 -- 172.21.15.192:0/4085817946 shutdown_connections 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 --2- 172.21.15.192:0/4085817946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440f58a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 -- 172.21.15.192:0/4085817946 >> 172.21.15.192:0/4085817946 conn(0x7f92440f0be0 msgr2=0x7f92440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 -- 172.21.15.192:0/4085817946 shutdown_connections 2024-02-20T13:50:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.219+0000 7f924969e700 1 -- 172.21.15.192:0/4085817946 wait complete. 2024-02-20T13:50:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f924969e700 1 Processor -- start 2024-02-20T13:50:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f924969e700 1 -- start start 2024-02-20T13:50:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f924969e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f924969e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440fb8e0 con 0x7f92440f5480 2024-02-20T13:50:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f9242ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.220+0000 7f9242ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40800/0 (socket says 172.21.15.192:40800) 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f9242ffd700 1 -- 172.21.15.192:0/1715933906 learned_addr learned my addr 172.21.15.192:0/1715933906 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f9242ffd700 1 -- 172.21.15.192:0/1715933906 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9238005040 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f9242ffd700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f92380089f0 tx=0x7f923800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9238009320 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92440fba80 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9238013ec0 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9238014d30 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.221+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92440fbf10 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.222+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9238014420 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.222+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92440591e0 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.222+0000 7f922b7fe700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 0x7f922c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.222+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9238042450 con 0x7f92440f5480 2024-02-20T13:50:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.222+0000 7f92427fc700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 0x7f922c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.223+0000 7f92427fc700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 0x7f922c0335a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f92340096a0 tx=0x7f9234006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:19.213 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.227+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f923801d7f0 con 0x7f92440f5480 2024-02-20T13:50:19.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.431+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92440faf70 con 0x7f92440f5480 2024-02-20T13:50:19.417 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.431+0000 7f922b7fe700 1 -- 172.21.15.192:0/1715933906 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f923801d7f0 con 0x7f92440f5480 2024-02-20T13:50:19.418 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:19.418 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 msgr2=0x7f922c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 0x7f922c0335a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f92340096a0 tx=0x7f9234006b40).stop 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 msgr2=0x7f92440fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f92380089f0 tx=0x7f923800b040).stop 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 shutdown_connections 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f922c0310e0 0x7f922c0335a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 --2- 172.21.15.192:0/1715933906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92440f5480 0x7f92440fd460 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 >> 172.21.15.192:0/1715933906 conn(0x7f92440f0be0 msgr2=0x7f924405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 shutdown_connections 2024-02-20T13:50:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:19.433+0000 7f924969e700 1 -- 172.21.15.192:0/1715933906 wait complete. 2024-02-20T13:50:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:19.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:17.414949+0000 mgr.smithi158.gmoobf (mgr.14156) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:20.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:19.432746+0000 mon.smithi158 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.192:0/1715933906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:21.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:21.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:21.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:19.415280+0000 mgr.smithi158.gmoobf (mgr.14156) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:23.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 -- 172.21.15.192:0/2324166315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 msgr2=0x7f3cb00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:23.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/2324166315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb00f58a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f3ca0009fb0 tx=0x7f3ca0009b30).stop 2024-02-20T13:50:23.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 -- 172.21.15.192:0/2324166315 shutdown_connections 2024-02-20T13:50:23.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/2324166315 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb00f58a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:23.106 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 -- 172.21.15.192:0/2324166315 >> 172.21.15.192:0/2324166315 conn(0x7f3cb00f0be0 msgr2=0x7f3cb00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 -- 172.21.15.192:0/2324166315 shutdown_connections 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.118+0000 7f3cb66c6700 1 -- 172.21.15.192:0/2324166315 wait complete. 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3cb66c6700 1 Processor -- start 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3cb66c6700 1 -- start start 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3cb66c6700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3cb66c6700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb0107300 con 0x7f3cb00f5480 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3caffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3caffff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36154/0 (socket says 172.21.15.192:36154) 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3caffff700 1 -- 172.21.15.192:0/1507412357 learned_addr learned my addr 172.21.15.192:0/1507412357 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.119+0000 7f3caffff700 1 -- 172.21.15.192:0/1507412357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0005040 con 0x7f3cb00f5480 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3caffff700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3ca0000c00 tx=0x7f3ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca00095f0 con 0x7f3cb00f5480 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb0060300 con 0x7f3cb00f5480 2024-02-20T13:50:23.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3ca0007e50 con 0x7f3cb00f5480 2024-02-20T13:50:23.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca0014e70 con 0x7f3cb00f5480 2024-02-20T13:50:23.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.120+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb010be00 con 0x7f3cb00f5480 2024-02-20T13:50:23.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.121+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3ca0025070 con 0x7f3cb00f5480 2024-02-20T13:50:23.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.121+0000 7f3cacff9700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 0x7f3c9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:23.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.121+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb0045bb0 con 0x7f3cb00f5480 2024-02-20T13:50:23.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.121+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3ca0041dd0 con 0x7f3cb00f5480 2024-02-20T13:50:23.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.121+0000 7f3caf7fe700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 0x7f3c9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:23.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.122+0000 7f3caf7fe700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 0x7f3c9c033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f3c980096a0 tx=0x7f3c98006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:23.112 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.126+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3ca00079c0 con 0x7f3cb00f5480 2024-02-20T13:50:23.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.331+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cb00fb060 con 0x7f3cb00f5480 2024-02-20T13:50:23.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.332+0000 7f3cacff9700 1 -- 172.21.15.192:0/1507412357 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ca000f300 con 0x7f3cb00f5480 2024-02-20T13:50:23.318 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:23.318 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 msgr2=0x7f3c9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 0x7f3c9c033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f3c980096a0 tx=0x7f3c98006b40).stop 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 msgr2=0x7f3cb010b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3ca0000c00 tx=0x7f3ca000b040).stop 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 shutdown_connections 2024-02-20T13:50:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3c9c031090 0x7f3c9c033550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:23.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 --2- 172.21.15.192:0/1507412357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3cb00f5480 0x7f3cb010b910 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:23.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 >> 172.21.15.192:0/1507412357 conn(0x7f3cb00f0be0 msgr2=0x7f3cb00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:23.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 shutdown_connections 2024-02-20T13:50:23.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:23.334+0000 7f3cb66c6700 1 -- 172.21.15.192:0/1507412357 wait complete. 2024-02-20T13:50:23.321 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:23.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:21.415669+0000 mgr.smithi158.gmoobf (mgr.14156) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:24.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:23.333475+0000 mon.smithi158 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.192:0/1507412357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:24.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:24.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:23.416047+0000 mgr.smithi158.gmoobf (mgr.14156) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:26.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 -- 172.21.15.192:0/2874267028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 msgr2=0x7f47a40f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:26.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 --2- 172.21.15.192:0/2874267028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a40f9160 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f478c009fb0 tx=0x7f478c009b30).stop 2024-02-20T13:50:26.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 -- 172.21.15.192:0/2874267028 shutdown_connections 2024-02-20T13:50:26.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 --2- 172.21.15.192:0/2874267028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a40f9160 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 -- 172.21.15.192:0/2874267028 >> 172.21.15.192:0/2874267028 conn(0x7f47a40f0be0 msgr2=0x7f47a40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 -- 172.21.15.192:0/2874267028 shutdown_connections 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 -- 172.21.15.192:0/2874267028 wait complete. 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.869+0000 7f47aa089700 1 Processor -- start 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47aa089700 1 -- start start 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47aa089700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47aa089700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4106c40 con 0x7f47a4060360 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47a37fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36164/0 (socket says 172.21.15.192:36164) 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47a37fe700 1 -- 172.21.15.192:0/1923807367 learned_addr learned my addr 172.21.15.192:0/1923807367 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47a37fe700 1 -- 172.21.15.192:0/1923807367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c005040 con 0x7f47a4060360 2024-02-20T13:50:26.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f47a37fe700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f478c009f80 tx=0x7f478c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:26.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.870+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478c009320 con 0x7f47a4060360 2024-02-20T13:50:26.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.871+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f478c013680 con 0x7f47a4060360 2024-02-20T13:50:26.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.871+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4106e40 con 0x7f47a4060360 2024-02-20T13:50:26.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.871+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478c01eda0 con 0x7f47a4060360 2024-02-20T13:50:26.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.871+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a41072d0 con 0x7f47a4060360 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.871+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f478c01e420 con 0x7f47a4060360 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.872+0000 7f479bfff700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 0x7f4790033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.872+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4045bb0 con 0x7f47a4060360 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.872+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f478c0423c0 con 0x7f47a4060360 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.872+0000 7f47a2ffd700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 0x7f4790033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:26.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.872+0000 7f47a2ffd700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 0x7f4790033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:26.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:26.877+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f478c01e700 con 0x7f47a4060360 2024-02-20T13:50:27.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.082+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a4108180 con 0x7f47a4060360 2024-02-20T13:50:27.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.083+0000 7f479bfff700 1 -- 172.21.15.192:0/1923807367 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f478c01cb10 con 0x7f47a4060360 2024-02-20T13:50:27.070 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:27.071 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:27.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 msgr2=0x7f4790033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 0x7f4790033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).stop 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 msgr2=0x7f47a4106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f478c009f80 tx=0x7f478c00b040).stop 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 shutdown_connections 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f4790031090 0x7f4790033550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 --2- 172.21.15.192:0/1923807367 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47a4060360 0x7f47a4106750 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 >> 172.21.15.192:0/1923807367 conn(0x7f47a40f0be0 msgr2=0x7f47a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 shutdown_connections 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:27.085+0000 7f47aa089700 1 -- 172.21.15.192:0/1923807367 wait complete. 2024-02-20T13:50:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:25.416282+0000 mgr.smithi158.gmoobf (mgr.14156) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:27.084456+0000 mon.smithi158 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.192:0/1923807367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:28.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:28.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:29.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:29 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:27.416645+0000 mgr.smithi158.gmoobf (mgr.14156) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- 172.21.15.192:0/154312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 msgr2=0x7fd6480f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 --2- 172.21.15.192:0/154312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd6480f6800 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fd644009fb0 tx=0x7fd644009b30).stop 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- 172.21.15.192:0/154312053 shutdown_connections 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 --2- 172.21.15.192:0/154312053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd6480f6800 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- 172.21.15.192:0/154312053 >> 172.21.15.192:0/154312053 conn(0x7fd6480f1b40 msgr2=0x7fd6480f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- 172.21.15.192:0/154312053 shutdown_connections 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- 172.21.15.192:0/154312053 wait complete. 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 Processor -- start 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- start start 2024-02-20T13:50:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.704+0000 7fd64ea74700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648106b10 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64da72700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64da72700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:36182/0 (socket says 172.21.15.192:36182) 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64da72700 1 -- 172.21.15.192:0/2130325505 learned_addr learned my addr 172.21.15.192:0/2130325505 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64da72700 1 -- 172.21.15.192:0/2130325505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644005040 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64da72700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd6440089f0 tx=0x7fd64400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd644009320 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd648106d10 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd644013ec0 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd644014d30 con 0x7fd6480f63e0 2024-02-20T13:50:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.705+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6481071a0 con 0x7fd6480f63e0 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.706+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fd6440095f0 con 0x7fd6480f63e0 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.706+0000 7fd63e7fc700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 0x7fd634033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.706+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd6440423c0 con 0x7fd6480f63e0 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.706+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd648045bb0 con 0x7fd6480f63e0 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.706+0000 7fd64d271700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 0x7fd634033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.707+0000 7fd64d271700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 0x7fd634033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd6380096a0 tx=0x7fd638006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.711+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd64401d980 con 0x7fd6480f63e0 2024-02-20T13:50:30.903 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.917+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6480fbfc0 con 0x7fd6480f63e0 2024-02-20T13:50:30.904 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.917+0000 7fd63e7fc700 1 -- 172.21.15.192:0/2130325505 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd64402b020 con 0x7fd6480f63e0 2024-02-20T13:50:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:30.904 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:30.905 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 msgr2=0x7fd634033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 0x7fd634033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd6380096a0 tx=0x7fd638006b40).stop 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 msgr2=0x7fd648106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd6440089f0 tx=0x7fd64400b040).stop 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 shutdown_connections 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fd634031090 0x7fd634033550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 --2- 172.21.15.192:0/2130325505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6480f63e0 0x7fd648106620 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 >> 172.21.15.192:0/2130325505 conn(0x7fd6480f1b40 msgr2=0x7fd6480f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 shutdown_connections 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:30.919+0000 7fd64ea74700 1 -- 172.21.15.192:0/2130325505 wait complete. 2024-02-20T13:50:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:31.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:29.416919+0000 mgr.smithi158.gmoobf (mgr.14156) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:31.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:30.918948+0000 mon.smithi158 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.192:0/2130325505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:32.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:32.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:33.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:33 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:31.417265+0000 mgr.smithi158.gmoobf (mgr.14156) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- 172.21.15.192:0/1137837586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 msgr2=0x7f45540f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 --2- 172.21.15.192:0/1137837586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45540f4c70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4544009fb0 tx=0x7f4544009b30).stop 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- 172.21.15.192:0/1137837586 shutdown_connections 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 --2- 172.21.15.192:0/1137837586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45540f4c70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- 172.21.15.192:0/1137837586 >> 172.21.15.192:0/1137837586 conn(0x7f45540f0390 msgr2=0x7f45540f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- 172.21.15.192:0/1137837586 shutdown_connections 2024-02-20T13:50:34.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- 172.21.15.192:0/1137837586 wait complete. 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 Processor -- start 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.645+0000 7f45592b9700 1 -- start start 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f45592b9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f45592b9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554106c90 con 0x7f45540f4850 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f4552ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f4552ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44528/0 (socket says 172.21.15.192:44528) 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f4552ffd700 1 -- 172.21.15.192:0/2617987894 learned_addr learned my addr 172.21.15.192:0/2617987894 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f4552ffd700 1 -- 172.21.15.192:0/2617987894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544005040 con 0x7f45540f4850 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f4552ffd700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f45440089f0 tx=0x7f454400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544009320 con 0x7f45540f4850 2024-02-20T13:50:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4544013ec0 con 0x7f45540f4850 2024-02-20T13:50:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.646+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4554106e90 con 0x7f45540f4850 2024-02-20T13:50:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544014d30 con 0x7f45540f4850 2024-02-20T13:50:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4554107320 con 0x7f45540f4850 2024-02-20T13:50:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f45440095f0 con 0x7f45540f4850 2024-02-20T13:50:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f453b7fe700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 0x7f453c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554045bb0 con 0x7f45540f4850 2024-02-20T13:50:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.647+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4544042410 con 0x7f45540f4850 2024-02-20T13:50:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.648+0000 7f45527fc700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 0x7f453c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.648+0000 7f45527fc700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 0x7f453c033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f45480096a0 tx=0x7f4548006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:34.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.652+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f454401d7f0 con 0x7f45540f4850 2024-02-20T13:50:34.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.859+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45540fc1d0 con 0x7f45540f4850 2024-02-20T13:50:34.845 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.859+0000 7f453b7fe700 1 -- 172.21.15.192:0/2617987894 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f454401d7f0 con 0x7f45540f4850 2024-02-20T13:50:34.846 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:34.846 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 msgr2=0x7f453c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 0x7f453c033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f45480096a0 tx=0x7f4548006b40).stop 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 msgr2=0x7f45541067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f45440089f0 tx=0x7f454400b040).stop 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 shutdown_connections 2024-02-20T13:50:34.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f453c031090 0x7f453c033550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:34.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 --2- 172.21.15.192:0/2617987894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45540f4850 0x7f45541067a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:34.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 >> 172.21.15.192:0/2617987894 conn(0x7f45540f0390 msgr2=0x7f45540f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:34.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 shutdown_connections 2024-02-20T13:50:34.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:34.861+0000 7f45592b9700 1 -- 172.21.15.192:0/2617987894 wait complete. 2024-02-20T13:50:34.848 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:35.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:33.417602+0000 mgr.smithi158.gmoobf (mgr.14156) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:35.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:34.860850+0000 mon.smithi158 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.192:0/2617987894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:36.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:36.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:37.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:37 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:35.417866+0000 mgr.smithi158.gmoobf (mgr.14156) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 -- 172.21.15.192:0/1619191295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 msgr2=0x7fc6940f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/1619191295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6940f5660 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fc684009fb0 tx=0x7fc684009b30).stop 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 -- 172.21.15.192:0/1619191295 shutdown_connections 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/1619191295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6940f5660 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 -- 172.21.15.192:0/1619191295 >> 172.21.15.192:0/1619191295 conn(0x7fc6940f09a0 msgr2=0x7fc6940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 -- 172.21.15.192:0/1619191295 shutdown_connections 2024-02-20T13:50:38.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.489+0000 7fc69a3a7700 1 -- 172.21.15.192:0/1619191295 wait complete. 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc69a3a7700 1 Processor -- start 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc69a3a7700 1 -- start start 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc69a3a7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc69a3a7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6941059d0 con 0x7fc6940f5240 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc6993a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc6993a5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:44560/0 (socket says 172.21.15.192:44560) 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.490+0000 7fc6993a5700 1 -- 172.21.15.192:0/3479430661 learned_addr learned my addr 172.21.15.192:0/3479430661 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc6993a5700 1 -- 172.21.15.192:0/3479430661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc684005040 con 0x7fc6940f5240 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc6993a5700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fc6840089f0 tx=0x7fc68400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc684009320 con 0x7fc6940f5240 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc694105bd0 con 0x7fc6940f5240 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc684013ec0 con 0x7fc6940f5240 2024-02-20T13:50:38.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc684014d30 con 0x7fc6940f5240 2024-02-20T13:50:38.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.491+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc694106060 con 0x7fc6940f5240 2024-02-20T13:50:38.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.492+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc6840095f0 con 0x7fc6940f5240 2024-02-20T13:50:38.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.492+0000 7fc689ffb700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 0x7fc680033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:38.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.492+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694045bb0 con 0x7fc6940f5240 2024-02-20T13:50:38.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.492+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc6840423d0 con 0x7fc6940f5240 2024-02-20T13:50:38.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.492+0000 7fc698ba4700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 0x7fc680033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:38.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.493+0000 7fc698ba4700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 0x7fc680033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fc6900096a0 tx=0x7fc690006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:38.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.497+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc68401d7f0 con 0x7fc6940f5240 2024-02-20T13:50:38.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.706+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6940fad30 con 0x7fc6940f5240 2024-02-20T13:50:38.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.707+0000 7fc689ffb700 1 -- 172.21.15.192:0/3479430661 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc68400fe40 con 0x7fc6940f5240 2024-02-20T13:50:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:38.693 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.708+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 msgr2=0x7fc680033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.708+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 0x7fc680033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fc6900096a0 tx=0x7fc690006b40).stop 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.708+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 msgr2=0x7fc6941054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.708+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fc6840089f0 tx=0x7fc68400b040).stop 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 shutdown_connections 2024-02-20T13:50:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc680031090 0x7fc680033550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:38.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 --2- 172.21.15.192:0/3479430661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6940f5240 0x7fc6941054e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:38.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 >> 172.21.15.192:0/3479430661 conn(0x7fc6940f09a0 msgr2=0x7fc6940f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:38.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 shutdown_connections 2024-02-20T13:50:38.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:38.709+0000 7fc69a3a7700 1 -- 172.21.15.192:0/3479430661 wait complete. 2024-02-20T13:50:38.696 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:39.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:37.418203+0000 mgr.smithi158.gmoobf (mgr.14156) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:39.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:39 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:38.708451+0000 mon.smithi158 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.192:0/3479430661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:40.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:40.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:41.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:41 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:39.418504+0000 mgr.smithi158.gmoobf (mgr.14156) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:42.515 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- 172.21.15.192:0/715701883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 msgr2=0x7f9db80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 --2- 172.21.15.192:0/715701883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db80f58a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f9db4009fb0 tx=0x7f9db4009b30).stop 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- 172.21.15.192:0/715701883 shutdown_connections 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 --2- 172.21.15.192:0/715701883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db80f58a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- 172.21.15.192:0/715701883 >> 172.21.15.192:0/715701883 conn(0x7f9db80f0be0 msgr2=0x7f9db80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- 172.21.15.192:0/715701883 shutdown_connections 2024-02-20T13:50:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- 172.21.15.192:0/715701883 wait complete. 2024-02-20T13:50:42.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 Processor -- start 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.529+0000 7f9dbf463700 1 -- start start 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbf463700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbf463700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db8060300 con 0x7f9db80f5480 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbd1ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbd1ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48958/0 (socket says 172.21.15.192:48958) 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbd1ff700 1 -- 172.21.15.192:0/2837539869 learned_addr learned my addr 172.21.15.192:0/2837539869 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbd1ff700 1 -- 172.21.15.192:0/2837539869 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db4005040 con 0x7f9db80f5480 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbd1ff700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f9db40089f0 tx=0x7f9db400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db4009320 con 0x7f9db80f5480 2024-02-20T13:50:42.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db8060500 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9db4013ec0 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.530+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db4014d30 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.531+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db8109bc0 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.532+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9db40095f0 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.532+0000 7f9dadffb700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.532+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9db4042390 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.532+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db80591e0 con 0x7f9db80f5480 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.532+0000 7f9dbc9fe700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.533+0000 7f9dbc9fe700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 0x7f9da4033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9da80096a0 tx=0x7f9da8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:42.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.537+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9db401d7f0 con 0x7f9db80f5480 2024-02-20T13:50:42.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.745+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9db810a1d0 con 0x7f9db80f5480 2024-02-20T13:50:42.732 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.746+0000 7f9dadffb700 1 -- 172.21.15.192:0/2837539869 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9db401d7f0 con 0x7f9db80f5480 2024-02-20T13:50:42.732 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:42.733 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 msgr2=0x7f9da4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 0x7f9da4033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9da80096a0 tx=0x7f9da8006b40).stop 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 msgr2=0x7f9db8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f9db40089f0 tx=0x7f9db400b040).stop 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 shutdown_connections 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:42.734 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 --2- 172.21.15.192:0/2837539869 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9db80f5480 0x7f9db8109510 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:42.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 >> 172.21.15.192:0/2837539869 conn(0x7f9db80f0be0 msgr2=0x7f9db80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:42.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 shutdown_connections 2024-02-20T13:50:42.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:42.748+0000 7f9dbf463700 1 -- 172.21.15.192:0/2837539869 wait complete. 2024-02-20T13:50:42.735 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:43.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:41.418815+0000 mgr.smithi158.gmoobf (mgr.14156) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:43.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:43 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:42.747598+0000 mon.smithi158 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.192:0/2837539869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:44.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:44.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:45.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:45 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:43.419192+0000 mgr.smithi158.gmoobf (mgr.14156) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 -- 172.21.15.192:0/3501766773 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 msgr2=0x7f5f840f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 --2- 172.21.15.192:0/3501766773 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f840f4c60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5f78009fb0 tx=0x7f5f78009b30).stop 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 -- 172.21.15.192:0/3501766773 shutdown_connections 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 --2- 172.21.15.192:0/3501766773 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f840f4c60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 -- 172.21.15.192:0/3501766773 >> 172.21.15.192:0/3501766773 conn(0x7f5f840f0360 msgr2=0x7f5f840f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 -- 172.21.15.192:0/3501766773 shutdown_connections 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 -- 172.21.15.192:0/3501766773 wait complete. 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.308+0000 7f5f88b01700 1 Processor -- start 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f88b01700 1 -- start start 2024-02-20T13:50:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f88b01700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f88b01700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84106cb0 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f8259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f8259c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48968/0 (socket says 172.21.15.192:48968) 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f8259c700 1 -- 172.21.15.192:0/2547027864 learned_addr learned my addr 172.21.15.192:0/2547027864 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f8259c700 1 -- 172.21.15.192:0/2547027864 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f78005040 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f8259c700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f5f780089f0 tx=0x7f5f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f78009320 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f84106eb0 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5f78013ec0 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f78014d30 con 0x7f5f840f4840 2024-02-20T13:50:46.296 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.309+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f84107340 con 0x7f5f840f4840 2024-02-20T13:50:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.310+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f5f780095f0 con 0x7f5f840f4840 2024-02-20T13:50:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.310+0000 7f5f72ffd700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 0x7f5f6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.310+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84045bb0 con 0x7f5f840f4840 2024-02-20T13:50:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.310+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5f78042390 con 0x7f5f840f4840 2024-02-20T13:50:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.311+0000 7f5f81d9b700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 0x7f5f6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:46.298 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.311+0000 7f5f81d9b700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 0x7f5f6c033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5f740096a0 tx=0x7f5f74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:46.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.315+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5f7800fe40 con 0x7f5f840f4840 2024-02-20T13:50:46.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.523+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f841081f0 con 0x7f5f840f4840 2024-02-20T13:50:46.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.523+0000 7f5f72ffd700 1 -- 172.21.15.192:0/2547027864 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f7800fe40 con 0x7f5f840f4840 2024-02-20T13:50:46.509 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:46.509 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 msgr2=0x7f5f6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 0x7f5f6c033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5f740096a0 tx=0x7f5f74006b40).stop 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 msgr2=0x7f5f841067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f5f780089f0 tx=0x7f5f7800b040).stop 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 shutdown_connections 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5f6c031090 0x7f5f6c033550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 --2- 172.21.15.192:0/2547027864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f840f4840 0x7f5f841067c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 >> 172.21.15.192:0/2547027864 conn(0x7f5f840f0360 msgr2=0x7f5f840f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 shutdown_connections 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:46.525+0000 7f5f88b01700 1 -- 172.21.15.192:0/2547027864 wait complete. 2024-02-20T13:50:46.511 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:46.782 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:45.419459+0000 mgr.smithi158.gmoobf (mgr.14156) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:47.782 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:47 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:46.523892+0000 mon.smithi158 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.192:0/2547027864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:48.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:48.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:48.782 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:47.419745+0000 mgr.smithi158.gmoobf (mgr.14156) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:50.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 -- 172.21.15.192:0/1140178053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 msgr2=0x7f3f1c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:50.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 --2- 172.21.15.192:0/1140178053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c0f69b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3f04004760 tx=0x7f3f04009ad0).stop 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 -- 172.21.15.192:0/1140178053 shutdown_connections 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 --2- 172.21.15.192:0/1140178053 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c0f69b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 -- 172.21.15.192:0/1140178053 >> 172.21.15.192:0/1140178053 conn(0x7f3f1c0f1cd0 msgr2=0x7f3f1c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 -- 172.21.15.192:0/1140178053 shutdown_connections 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.240+0000 7f3f22068700 1 -- 172.21.15.192:0/1140178053 wait complete. 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f22068700 1 Processor -- start 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f22068700 1 -- start start 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f22068700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f22068700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c106cb0 con 0x7f3f1c0f6590 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f1b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f1b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:48984/0 (socket says 172.21.15.192:48984) 2024-02-20T13:50:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.241+0000 7f3f1b7fe700 1 -- 172.21.15.192:0/2420479258 learned_addr learned my addr 172.21.15.192:0/2420479258 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f1b7fe700 1 -- 172.21.15.192:0/2420479258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04009d70 con 0x7f3f1c0f6590 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f1b7fe700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3f04013f10 tx=0x7f3f04009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0400e040 con 0x7f3f1c0f6590 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c106eb0 con 0x7f3f1c0f6590 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3f04007df0 con 0x7f3f1c0f6590 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f04014d60 con 0x7f3f1c0f6590 2024-02-20T13:50:50.229 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.242+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c107340 con 0x7f3f1c0f6590 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.243+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3f04007960 con 0x7f3f1c0f6590 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.243+0000 7f3f13fff700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 0x7f3f08033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.243+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3f04042580 con 0x7f3f1c0f6590 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.243+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c045bb0 con 0x7f3f1c0f6590 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.243+0000 7f3f1affd700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 0x7f3f08033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:50.230 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.244+0000 7f3f1affd700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 0x7f3f08033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3f0c0096a0 tx=0x7f3f0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:50.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.248+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3f04014520 con 0x7f3f1c0f6590 2024-02-20T13:50:50.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.453+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f1c107fc0 con 0x7f3f1c0f6590 2024-02-20T13:50:50.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.454+0000 7f3f13fff700 1 -- 172.21.15.192:0/2420479258 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f04012330 con 0x7f3f1c0f6590 2024-02-20T13:50:50.439 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:50.439 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:50.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 msgr2=0x7f3f08033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:50.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 0x7f3f08033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3f0c0096a0 tx=0x7f3f0c006b40).stop 2024-02-20T13:50:50.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 msgr2=0x7f3f1c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:50.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3f04013f10 tx=0x7f3f04009ad0).stop 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 shutdown_connections 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f3f08031090 0x7f3f08033550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 --2- 172.21.15.192:0/2420479258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f1c0f6590 0x7f3f1c1067c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 >> 172.21.15.192:0/2420479258 conn(0x7f3f1c0f1cd0 msgr2=0x7f3f1c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 shutdown_connections 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:50.455+0000 7f3f22068700 1 -- 172.21.15.192:0/2420479258 wait complete. 2024-02-20T13:50:50.441 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:51.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:49.420071+0000 mgr.smithi158.gmoobf (mgr.14156) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:51.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:51 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:50.454005+0000 mon.smithi158 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.192:0/2420479258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:52.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:52.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:53.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:53 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:51.420435+0000 mgr.smithi158.gmoobf (mgr.14156) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:54.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 -- 172.21.15.192:0/2233878657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 msgr2=0x7fa6b00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:54.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 --2- 172.21.15.192:0/2233878657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b00f69b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa6a4004760 tx=0x7fa6a4009ad0).stop 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 -- 172.21.15.192:0/2233878657 shutdown_connections 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 --2- 172.21.15.192:0/2233878657 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b00f69b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 -- 172.21.15.192:0/2233878657 >> 172.21.15.192:0/2233878657 conn(0x7fa6b00f1cd0 msgr2=0x7fa6b00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 -- 172.21.15.192:0/2233878657 shutdown_connections 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.319+0000 7fa6b6599700 1 -- 172.21.15.192:0/2233878657 wait complete. 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6b6599700 1 Processor -- start 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6b6599700 1 -- start start 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6b6599700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6b6599700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0069f00 con 0x7fa6b00f6590 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53650/0 (socket says 172.21.15.192:53650) 2024-02-20T13:50:54.307 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.320+0000 7fa6affff700 1 -- 172.21.15.192:0/70895502 learned_addr learned my addr 172.21.15.192:0/70895502 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6affff700 1 -- 172.21.15.192:0/70895502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4009d70 con 0x7fa6b00f6590 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6affff700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fa6a4013f10 tx=0x7fa6a4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a400e040 con 0x7fa6b00f6590 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b006a0a0 con 0x7fa6b00f6590 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa6a4007df0 con 0x7fa6b00f6590 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4014d60 con 0x7fa6b00f6590 2024-02-20T13:50:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.321+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b006a530 con 0x7fa6b00f6590 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.322+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa6a401f460 con 0x7fa6b00f6590 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.322+0000 7fa6acff9700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 0x7fa698033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.322+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b00591e0 con 0x7fa6b00f6590 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.322+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa6a4042500 con 0x7fa6b00f6590 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.322+0000 7fa6af7fe700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 0x7fa698033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:54.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.323+0000 7fa6af7fe700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 0x7fa698033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:54.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.327+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa6a4014520 con 0x7fa6b00f6590 2024-02-20T13:50:54.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.531+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6b006acd0 con 0x7fa6b00f6590 2024-02-20T13:50:54.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.531+0000 7fa6acff9700 1 -- 172.21.15.192:0/70895502 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa6a4026760 con 0x7fa6b00f6590 2024-02-20T13:50:54.517 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:54.517 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 msgr2=0x7fa698033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 0x7fa698033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).stop 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 msgr2=0x7fa6b006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fa6a4013f10 tx=0x7fa6a4009ad0).stop 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 shutdown_connections 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa698031090 0x7fa698033550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:54.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 --2- 172.21.15.192:0/70895502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6b00f6590 0x7fa6b006ba80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:54.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 >> 172.21.15.192:0/70895502 conn(0x7fa6b00f1cd0 msgr2=0x7fa6b00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:54.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 shutdown_connections 2024-02-20T13:50:54.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:54.533+0000 7fa6b6599700 1 -- 172.21.15.192:0/70895502 wait complete. 2024-02-20T13:50:54.519 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:55.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:53.420772+0000 mgr.smithi158.gmoobf (mgr.14156) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:55.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:55 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:54.532018+0000 mon.smithi158 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.192:0/70895502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:56.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:56.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:50:56.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:55.468987+0000 mon.smithi158 (mon.0) 290 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:50:56.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:55.469190+0000 mon.smithi158 (mon.0) 291 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:50:57.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:57 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:55.421154+0000 mgr.smithi158.gmoobf (mgr.14156) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:57.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.988+0000 7fb4d1570700 1 -- 172.21.15.192:0/1600337954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 msgr2=0x7fb4cc0f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.988+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1600337954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc0f8fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fb4bc009fb0 tx=0x7fb4bc009b30).stop 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.988+0000 7fb4d1570700 1 -- 172.21.15.192:0/1600337954 shutdown_connections 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.988+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1600337954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc0f8fb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.988+0000 7fb4d1570700 1 -- 172.21.15.192:0/1600337954 >> 172.21.15.192:0/1600337954 conn(0x7fb4cc0f0a30 msgr2=0x7fb4cc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 -- 172.21.15.192:0/1600337954 shutdown_connections 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 -- 172.21.15.192:0/1600337954 wait complete. 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 Processor -- start 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 -- start start 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.989+0000 7fb4d1570700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc106aa0 con 0x7fb4cc060360 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4cbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4cbfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53676/0 (socket says 172.21.15.192:53676) 2024-02-20T13:50:57.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4cbfff700 1 -- 172.21.15.192:0/1663517623 learned_addr learned my addr 172.21.15.192:0/1663517623 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4cbfff700 1 -- 172.21.15.192:0/1663517623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc005040 con 0x7fb4cc060360 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4cbfff700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb4bc009f80 tx=0x7fb4bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4bc009320 con 0x7fb4cc060360 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc106ca0 con 0x7fb4cc060360 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb4bc013680 con 0x7fb4cc060360 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4bc01eda0 con 0x7fb4cc060360 2024-02-20T13:50:57.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.990+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc107130 con 0x7fb4cc060360 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.991+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fb4bc01e420 con 0x7fb4cc060360 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.991+0000 7fb4c8ff9700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 0x7fb4b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.991+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc045bb0 con 0x7fb4cc060360 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.991+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb4bc0423c0 con 0x7fb4cc060360 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.992+0000 7fb4cb7fe700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 0x7fb4b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:50:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.992+0000 7fb4cb7fe700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 0x7fb4b4033550 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb4c00096a0 tx=0x7fb4c0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:50:57.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:57.996+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb4bc01e700 con 0x7fb4cc060360 2024-02-20T13:50:58.187 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.201+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4cc0fbeb0 con 0x7fb4cc060360 2024-02-20T13:50:58.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.202+0000 7fb4c8ff9700 1 -- 172.21.15.192:0/1663517623 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb4bc01cb10 con 0x7fb4cc060360 2024-02-20T13:50:58.189 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:50:58.189 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 msgr2=0x7fb4b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 0x7fb4b4033550 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb4c00096a0 tx=0x7fb4c0006b40).stop 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 msgr2=0x7fb4cc1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb4bc009f80 tx=0x7fb4bc00b040).stop 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 shutdown_connections 2024-02-20T13:50:58.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fb4b4031090 0x7fb4b4033550 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:58.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 --2- 172.21.15.192:0/1663517623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4cc060360 0x7fb4cc1065b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:50:58.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 >> 172.21.15.192:0/1663517623 conn(0x7fb4cc0f0a30 msgr2=0x7fb4cc0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:50:58.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 shutdown_connections 2024-02-20T13:50:58.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:50:58.204+0000 7fb4d1570700 1 -- 172.21.15.192:0/1663517623 wait complete. 2024-02-20T13:50:58.191 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:50:59.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:57.421551+0000 mgr.smithi158.gmoobf (mgr.14156) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:50:59.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:50:59 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:50:58.203887+0000 mon.smithi158 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.192:0/1663517623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:50:59.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:50:59.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:01.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:50:59.421878+0000 mgr.smithi158.gmoobf (mgr.14156) 313 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:01.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- 172.21.15.192:0/3954073705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 msgr2=0x7f0f240f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:01.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/3954073705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f240f56f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0f18009fb0 tx=0x7f0f18009b30).stop 2024-02-20T13:51:01.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- 172.21.15.192:0/3954073705 shutdown_connections 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/3954073705 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f240f56f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- 172.21.15.192:0/3954073705 >> 172.21.15.192:0/3954073705 conn(0x7f0f240f0a30 msgr2=0x7f0f240f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- 172.21.15.192:0/3954073705 shutdown_connections 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- 172.21.15.192:0/3954073705 wait complete. 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 Processor -- start 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.696+0000 7f0f29b3d700 1 -- start start 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f29b3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f29b3d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24060300 con 0x7f0f240f52d0 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f28b3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f28b3b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:53690/0 (socket says 172.21.15.192:53690) 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f28b3b700 1 -- 172.21.15.192:0/2730345272 learned_addr learned my addr 172.21.15.192:0/2730345272 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f28b3b700 1 -- 172.21.15.192:0/2730345272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18005040 con 0x7f0f240f52d0 2024-02-20T13:51:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f28b3b700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f0f18000c00 tx=0x7f0f1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:01.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f180095f0 con 0x7f0f240f52d0 2024-02-20T13:51:01.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f24060500 con 0x7f0f240f52d0 2024-02-20T13:51:01.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0f18007e50 con 0x7f0f240f52d0 2024-02-20T13:51:01.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f18014e70 con 0x7f0f240f52d0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.697+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f241099e0 con 0x7f0f240f52d0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.698+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f0f18025070 con 0x7f0f240f52d0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.698+0000 7f0f217fa700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 0x7f0f14033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.698+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0f18041d80 con 0x7f0f240f52d0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.699+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f240591e0 con 0x7f0f240f52d0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.699+0000 7f0f23fff700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 0x7f0f14033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.699+0000 7f0f23fff700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 0x7f0f14033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0f100096a0 tx=0x7f0f10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:01.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.703+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0f18026350 con 0x7f0f240f52d0 2024-02-20T13:51:01.895 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.909+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f24045bb0 con 0x7f0f240f52d0 2024-02-20T13:51:01.896 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.910+0000 7f0f217fa700 1 -- 172.21.15.192:0/2730345272 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f1802d090 con 0x7f0f240f52d0 2024-02-20T13:51:01.896 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:01.896 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.911+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 msgr2=0x7f0f14033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.911+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 0x7f0f14033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0f100096a0 tx=0x7f0f10006b40).stop 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.911+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 msgr2=0x7f0f24109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.911+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f0f18000c00 tx=0x7f0f1800b040).stop 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 shutdown_connections 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f0f14031090 0x7f0f14033550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 --2- 172.21.15.192:0/2730345272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f240f52d0 0x7f0f24109330 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 >> 172.21.15.192:0/2730345272 conn(0x7f0f240f0a30 msgr2=0x7f0f240f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 shutdown_connections 2024-02-20T13:51:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:01.912+0000 7f0f29b3d700 1 -- 172.21.15.192:0/2730345272 wait complete. 2024-02-20T13:51:01.899 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:02.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:01.911597+0000 mon.smithi158 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.192:0/2730345272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:03.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:03 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:01.422222+0000 mgr.smithi158.gmoobf (mgr.14156) 314 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:03.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:03.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 -- 172.21.15.192:0/2537534047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 msgr2=0x7f37fc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 --2- 172.21.15.192:0/2537534047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc0f58a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f37f0009fb0 tx=0x7f37f0009b30).stop 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 -- 172.21.15.192:0/2537534047 shutdown_connections 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 --2- 172.21.15.192:0/2537534047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc0f58a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 -- 172.21.15.192:0/2537534047 >> 172.21.15.192:0/2537534047 conn(0x7f37fc0f0be0 msgr2=0x7f37fc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 -- 172.21.15.192:0/2537534047 shutdown_connections 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 -- 172.21.15.192:0/2537534047 wait complete. 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.534+0000 7f3803227700 1 Processor -- start 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3803227700 1 -- start start 2024-02-20T13:51:05.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3803227700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3803227700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc106ce0 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3800fc3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3800fc3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55066/0 (socket says 172.21.15.192:55066) 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3800fc3700 1 -- 172.21.15.192:0/146035350 learned_addr learned my addr 172.21.15.192:0/146035350 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3800fc3700 1 -- 172.21.15.192:0/146035350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f0005040 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.535+0000 7f3800fc3700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f37f00089f0 tx=0x7f37f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.536+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f0009320 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.536+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc106ee0 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.536+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f37f0013ec0 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.536+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f0014d30 con 0x7f37fc0f5480 2024-02-20T13:51:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.536+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc107370 con 0x7f37fc0f5480 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.537+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc045bb0 con 0x7f37fc0f5480 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.537+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f37f00095f0 con 0x7f37fc0f5480 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.537+0000 7f37f97fa700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.537+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f37f0042490 con 0x7f37fc0f5480 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.537+0000 7f37fbfff700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:05.524 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.538+0000 7f37fbfff700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 0x7f37ec033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f37e80096a0 tx=0x7f37e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:05.527 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.542+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f37f001d7f0 con 0x7f37fc0f5480 2024-02-20T13:51:05.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:03.422614+0000 mgr.smithi158.gmoobf (mgr.14156) 315 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.747+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37fc0faf70 con 0x7f37fc0f5480 2024-02-20T13:51:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.748+0000 7f37f97fa700 1 -- 172.21.15.192:0/146035350 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37f000fe40 con 0x7f37fc0f5480 2024-02-20T13:51:05.734 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:05.734 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:05.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 msgr2=0x7f37ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:05.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 0x7f37ec033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f37e80096a0 tx=0x7f37e8006b40).stop 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 msgr2=0x7f37fc1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f37f00089f0 tx=0x7f37f000b040).stop 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 shutdown_connections 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 --2- 172.21.15.192:0/146035350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37fc0f5480 0x7f37fc1067f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 >> 172.21.15.192:0/146035350 conn(0x7f37fc0f0be0 msgr2=0x7f37fc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.749+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 shutdown_connections 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:05.750+0000 7f3803227700 1 -- 172.21.15.192:0/146035350 wait complete. 2024-02-20T13:51:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:06.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:06 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:05.749134+0000 mon.smithi158 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.192:0/146035350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:07.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:07.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:07.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:05.422997+0000 mgr.smithi158.gmoobf (mgr.14156) 316 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 -- 172.21.15.192:0/1572037324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 msgr2=0x7f79e00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 --2- 172.21.15.192:0/1572037324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00f58a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f79d4009fb0 tx=0x7f79d4009b30).stop 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 -- 172.21.15.192:0/1572037324 shutdown_connections 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 --2- 172.21.15.192:0/1572037324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00f58a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 -- 172.21.15.192:0/1572037324 >> 172.21.15.192:0/1572037324 conn(0x7f79e00f0be0 msgr2=0x7f79e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 -- 172.21.15.192:0/1572037324 shutdown_connections 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.252+0000 7f79e566b700 1 -- 172.21.15.192:0/1572037324 wait complete. 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79e566b700 1 Processor -- start 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79e566b700 1 -- start start 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79e566b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79e566b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00fb8e0 con 0x7f79e00f5480 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79deffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79deffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55074/0 (socket says 172.21.15.192:55074) 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79deffd700 1 -- 172.21.15.192:0/413234266 learned_addr learned my addr 172.21.15.192:0/413234266 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79deffd700 1 -- 172.21.15.192:0/413234266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4005040 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.253+0000 7f79deffd700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f79d40089f0 tx=0x7f79d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79d4009320 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e00fba80 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f79d4013ec0 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79d4014d30 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e00fbf10 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.254+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f79d40095f0 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.255+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e00591e0 con 0x7f79e00f5480 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.255+0000 7f79c77fe700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 0x7f79c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.255+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f79d40423d0 con 0x7f79e00f5480 2024-02-20T13:51:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.255+0000 7f79de7fc700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 0x7f79c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.255+0000 7f79de7fc700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 0x7f79c8033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f79d00096a0 tx=0x7f79d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.259+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f79d401d7f0 con 0x7f79e00f5480 2024-02-20T13:51:09.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.466+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79e0045bb0 con 0x7f79e00f5480 2024-02-20T13:51:09.453 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.467+0000 7f79c77fe700 1 -- 172.21.15.192:0/413234266 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f79d400fe40 con 0x7f79e00f5480 2024-02-20T13:51:09.453 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:09.453 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:09.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 msgr2=0x7f79c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:09.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 0x7f79c8033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f79d00096a0 tx=0x7f79d0006b40).stop 2024-02-20T13:51:09.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 msgr2=0x7f79e00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:09.454 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f79d40089f0 tx=0x7f79d400b040).stop 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 shutdown_connections 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f79c8031090 0x7f79c8033550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 --2- 172.21.15.192:0/413234266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79e00f5480 0x7f79e00fd460 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 >> 172.21.15.192:0/413234266 conn(0x7f79e00f0be0 msgr2=0x7f79e005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 shutdown_connections 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:09.468+0000 7f79e566b700 1 -- 172.21.15.192:0/413234266 wait complete. 2024-02-20T13:51:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:09.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:09 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:07.423350+0000 mgr.smithi158.gmoobf (mgr.14156) 317 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:10.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:09.468129+0000 mon.smithi158 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.192:0/413234266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:11.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:11.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:11.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:11 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:09.423794+0000 mgr.smithi158.gmoobf (mgr.14156) 318 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:13.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 -- 172.21.15.192:0/2949876173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 msgr2=0x7fa2d80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 --2- 172.21.15.192:0/2949876173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d80f69b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa2cc004760 tx=0x7fa2cc009ad0).stop 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 -- 172.21.15.192:0/2949876173 shutdown_connections 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 --2- 172.21.15.192:0/2949876173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d80f69b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 -- 172.21.15.192:0/2949876173 >> 172.21.15.192:0/2949876173 conn(0x7fa2d80f1cd0 msgr2=0x7fa2d80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 -- 172.21.15.192:0/2949876173 shutdown_connections 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.080+0000 7fa2df115700 1 -- 172.21.15.192:0/2949876173 wait complete. 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2df115700 1 Processor -- start 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2df115700 1 -- start start 2024-02-20T13:51:13.080 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2df115700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2df115700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8106cb0 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2dceb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2dceb1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41910/0 (socket says 172.21.15.192:41910) 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2dceb1700 1 -- 172.21.15.192:0/1195379734 learned_addr learned my addr 172.21.15.192:0/1195379734 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2dceb1700 1 -- 172.21.15.192:0/1195379734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc009d70 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.081+0000 7fa2dceb1700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa2cc004170 tx=0x7fa2cc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.082+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2cc00e040 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.082+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2cc008ef0 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.082+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d8106eb0 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.082+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2cc014c90 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.082+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d8107340 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa2cc014420 con 0x7fa2d80f6590 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2d57fa700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 0x7fa2c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:13.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d8045bb0 con 0x7fa2d80f6590 2024-02-20T13:51:13.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2cc042210 con 0x7fa2d80f6590 2024-02-20T13:51:13.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2d7fff700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 0x7fa2c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:13.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.083+0000 7fa2d7fff700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 0x7fa2c8033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa2c40096a0 tx=0x7fa2c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:13.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.088+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2cc012d00 con 0x7fa2d80f6590 2024-02-20T13:51:13.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.295+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2d81081c0 con 0x7fa2d80f6590 2024-02-20T13:51:13.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.296+0000 7fa2d57fa700 1 -- 172.21.15.192:0/1195379734 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa2cc012b20 con 0x7fa2d80f6590 2024-02-20T13:51:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:13.282 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:13.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 msgr2=0x7fa2c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:13.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 0x7fa2c8033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa2c40096a0 tx=0x7fa2c4006b40).stop 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 msgr2=0x7fa2d81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa2cc004170 tx=0x7fa2cc009ad0).stop 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 shutdown_connections 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2c8031090 0x7fa2c8033550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 --2- 172.21.15.192:0/1195379734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2d80f6590 0x7fa2d81067c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.297+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 >> 172.21.15.192:0/1195379734 conn(0x7fa2d80f1cd0 msgr2=0x7fa2d80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.298+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 shutdown_connections 2024-02-20T13:51:13.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:13.298+0000 7fa2df115700 1 -- 172.21.15.192:0/1195379734 wait complete. 2024-02-20T13:51:13.287 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:13.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:11.424145+0000 mgr.smithi158.gmoobf (mgr.14156) 319 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:14.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:14 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:13.297150+0000 mon.smithi158 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.192:0/1195379734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:15.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:15.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:15.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:15 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:13.424435+0000 mgr.smithi158.gmoobf (mgr.14156) 320 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 -- 172.21.15.192:0/2347228137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 msgr2=0x7fdd3c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 --2- 172.21.15.192:0/2347228137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c0f69b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fdd30004760 tx=0x7fdd30009ad0).stop 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 -- 172.21.15.192:0/2347228137 shutdown_connections 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 --2- 172.21.15.192:0/2347228137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c0f69b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 -- 172.21.15.192:0/2347228137 >> 172.21.15.192:0/2347228137 conn(0x7fdd3c0f1cd0 msgr2=0x7fdd3c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 -- 172.21.15.192:0/2347228137 shutdown_connections 2024-02-20T13:51:17.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.080+0000 7fdd42f57700 1 -- 172.21.15.192:0/2347228137 wait complete. 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd42f57700 1 Processor -- start 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd42f57700 1 -- start start 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd42f57700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd42f57700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c069f00 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd40cf3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd40cf3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41920/0 (socket says 172.21.15.192:41920) 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd40cf3700 1 -- 172.21.15.192:0/883633792 learned_addr learned my addr 172.21.15.192:0/883633792 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.081+0000 7fdd40cf3700 1 -- 172.21.15.192:0/883633792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd30009d70 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd40cf3700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fdd30004170 tx=0x7fdd30009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3000e040 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c06a0a0 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdd30008ef0 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd30014c90 con 0x7fdd3c0f6590 2024-02-20T13:51:17.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.082+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c06a530 con 0x7fdd3c0f6590 2024-02-20T13:51:17.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.083+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdd30014420 con 0x7fdd3c0f6590 2024-02-20T13:51:17.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.083+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c0591e0 con 0x7fdd3c0f6590 2024-02-20T13:51:17.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.083+0000 7fdd397fa700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 0x7fdd2c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:17.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.083+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdd30042210 con 0x7fdd3c0f6590 2024-02-20T13:51:17.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.083+0000 7fdd3bfff700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 0x7fdd2c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:17.071 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.084+0000 7fdd3bfff700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 0x7fdd2c033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fdd280096a0 tx=0x7fdd28006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:17.074 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.088+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdd30026300 con 0x7fdd3c0f6590 2024-02-20T13:51:17.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.296+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd3c045bb0 con 0x7fdd3c0f6590 2024-02-20T13:51:17.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.297+0000 7fdd397fa700 1 -- 172.21.15.192:0/883633792 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd30012b20 con 0x7fdd3c0f6590 2024-02-20T13:51:17.283 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:17.283 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 msgr2=0x7fdd2c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 0x7fdd2c033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fdd280096a0 tx=0x7fdd28006b40).stop 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 msgr2=0x7fdd3c06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fdd30004170 tx=0x7fdd30009ad0).stop 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 shutdown_connections 2024-02-20T13:51:17.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fdd2c031090 0x7fdd2c033550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:17.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 --2- 172.21.15.192:0/883633792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd3c0f6590 0x7fdd3c06ba80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:17.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 >> 172.21.15.192:0/883633792 conn(0x7fdd3c0f1cd0 msgr2=0x7fdd3c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:17.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 shutdown_connections 2024-02-20T13:51:17.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:17.299+0000 7fdd42f57700 1 -- 172.21.15.192:0/883633792 wait complete. 2024-02-20T13:51:17.286 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:17.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:15.424712+0000 mgr.smithi158.gmoobf (mgr.14156) 321 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:18.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:17.298686+0000 mon.smithi158 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.192:0/883633792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:18.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:18.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:19.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:19 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:17.425096+0000 mgr.smithi158.gmoobf (mgr.14156) 322 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/1885638989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 msgr2=0x7f9df40f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/1885638989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df40f4c60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f9df0009fb0 tx=0x7f9df0009b30).stop 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/1885638989 shutdown_connections 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/1885638989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df40f4c60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/1885638989 >> 172.21.15.192:0/1885638989 conn(0x7f9df40f0360 msgr2=0x7f9df40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/1885638989 shutdown_connections 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/1885638989 wait complete. 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.944+0000 7f9dfb7fb700 1 Processor -- start 2024-02-20T13:51:20.931 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9dfb7fb700 1 -- start start 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9dfb7fb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9dfb7fb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4106cb0 con 0x7f9df40f4840 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9df9597700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9df9597700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:41928/0 (socket says 172.21.15.192:41928) 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9df9597700 1 -- 172.21.15.192:0/677855518 learned_addr learned my addr 172.21.15.192:0/677855518 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:20.932 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9df9597700 1 -- 172.21.15.192:0/677855518 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0005040 con 0x7f9df40f4840 2024-02-20T13:51:20.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9df9597700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f9df00089f0 tx=0x7f9df000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:20.933 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df0009320 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9df0013ec0 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df4106eb0 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.945+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df0014d30 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.946+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df4107340 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f9df00095f0 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df4045bb0 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9de9ffb700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 0x7f9de00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9df0042450 con 0x7f9df40f4840 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9df8d96700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 0x7f9de00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.947+0000 7f9df8d96700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 0x7f9de00335a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9de40096a0 tx=0x7f9de4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:20.937 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:20.952+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9df00263a0 con 0x7f9df40f4840 2024-02-20T13:51:21.143 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.158+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9df41081f0 con 0x7f9df40f4840 2024-02-20T13:51:21.144 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.158+0000 7f9de9ffb700 1 -- 172.21.15.192:0/677855518 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9df001d7f0 con 0x7f9df40f4840 2024-02-20T13:51:21.144 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:21.144 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:21.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 msgr2=0x7f9de00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:21.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 0x7f9de00335a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9de40096a0 tx=0x7f9de4006b40).stop 2024-02-20T13:51:21.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 msgr2=0x7f9df41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:21.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f9df00089f0 tx=0x7f9df000b040).stop 2024-02-20T13:51:21.146 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 shutdown_connections 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9de00310e0 0x7f9de00335a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 --2- 172.21.15.192:0/677855518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9df40f4840 0x7f9df41067c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 >> 172.21.15.192:0/677855518 conn(0x7f9df40f0360 msgr2=0x7f9df40f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 shutdown_connections 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:21.160+0000 7f9dfb7fb700 1 -- 172.21.15.192:0/677855518 wait complete. 2024-02-20T13:51:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:21.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:19.425438+0000 mgr.smithi158.gmoobf (mgr.14156) 323 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:21.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:21 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:21.159687+0000 mon.smithi158 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.192:0/677855518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:22.283 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.055400+0000 mon.smithi158 (mon.0) 299 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:51:22.283 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.056971+0000 mon.smithi158 (mon.0) 300 : audit [DBG] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:51:22.283 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.057820+0000 mon.smithi158 (mon.0) 301 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T13:51:22.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:22.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:23.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:21.425824+0000 mgr.smithi158.gmoobf (mgr.14156) 324 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:23.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:51:22.053997+0000 mgr.smithi158.gmoobf (mgr.14156) 325 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon prometheus.smithi158 ... 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:49:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:51:23.533 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: yield (conn, connr) 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:51:23.534 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: code, '\n'.join(err))) 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon prometheus.smithi158 ... 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:49:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:51:23.535 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:51:22.054295+0000 mgr.smithi158.gmoobf (mgr.14156) 326 : cephadm [ERR] Failed while placing prometheus.smithi158 on smithi158: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: /bin/podman: stderr Error: no such container ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-prometheus.smithi158 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Deploy daemon prometheus.smithi158 ... 2024-02-20T13:51:23.536 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi158 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:49:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr time="2024-02-20T13:50:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: Traceback (most recent call last): 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: main() 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:51:23.537 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: r = ctx.func(ctx) 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: return func(ctx) 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: File "/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:51:23.538 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: raise RuntimeError('uid/gid not found') 2024-02-20T13:51:23.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: RuntimeError: uid/gid not found 2024-02-20T13:51:23.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.294510+0000 mon.smithi158 (mon.0) 302 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:23.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.491007+0000 mon.smithi158 (mon.0) 303 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:23.539 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:23 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:22.718047+0000 mon.smithi158 (mon.0) 304 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:23.426064+0000 mgr.smithi158.gmoobf (mgr.14156) 327 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:24.333724+0000 mon.smithi158 (mon.0) 305 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:24.577860+0000 mon.smithi158 (mon.0) 306 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-20T13:51:25.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:25 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:24.623148+0000 mon.smithi158 (mon.0) 307 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:26.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:51:24.578375+0000 mgr.smithi158.gmoobf (mgr.14156) 328 : cephadm [INF] Updating smithi158:/etc/ceph/ceph.conf 2024-02-20T13:51:26.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:51:24.595273+0000 mgr.smithi158.gmoobf (mgr.14156) 329 : cephadm [INF] Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-20T13:51:26.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:25.476847+0000 mon.smithi158 (mon.0) 308 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- 172.21.15.192:0/546542090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 msgr2=0x7f30140f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 --2- 172.21.15.192:0/546542090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f30140f69b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3004009fb0 tx=0x7f3004009b30).stop 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- 172.21.15.192:0/546542090 shutdown_connections 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 --2- 172.21.15.192:0/546542090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f30140f69b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- 172.21.15.192:0/546542090 >> 172.21.15.192:0/546542090 conn(0x7f30140f1cf0 msgr2=0x7f30140f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- 172.21.15.192:0/546542090 shutdown_connections 2024-02-20T13:51:26.760 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- 172.21.15.192:0/546542090 wait complete. 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 Processor -- start 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.774+0000 7f3019084700 1 -- start start 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3019084700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3019084700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3014069f00 con 0x7f30140f6590 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3012d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3012d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:42812/0 (socket says 172.21.15.192:42812) 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3012d9d700 1 -- 172.21.15.192:0/396699628 learned_addr learned my addr 172.21.15.192:0/396699628 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3012d9d700 1 -- 172.21.15.192:0/396699628 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004005040 con 0x7f30140f6590 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3012d9d700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f30040089f0 tx=0x7f300400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3004009320 con 0x7f30140f6590 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3004013ec0 con 0x7f30140f6590 2024-02-20T13:51:26.762 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301406a0a0 con 0x7f30140f6590 2024-02-20T13:51:26.763 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3004014d30 con 0x7f30140f6590 2024-02-20T13:51:26.763 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.775+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301406a530 con 0x7f30140f6590 2024-02-20T13:51:26.763 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.776+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f3004014420 con 0x7f30140f6590 2024-02-20T13:51:26.764 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.777+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30140591e0 con 0x7f30140f6590 2024-02-20T13:51:26.764 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.777+0000 7f2ffb7fe700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 0x7f2ffc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:26.764 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.777+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3004042350 con 0x7f30140f6590 2024-02-20T13:51:26.764 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.777+0000 7f301259c700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 0x7f2ffc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:26.764 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.777+0000 7f301259c700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 0x7f2ffc0335a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f30080096a0 tx=0x7f3008006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:26.767 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.781+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f30040268c0 con 0x7f30140f6590 2024-02-20T13:51:26.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.992+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30140fc080 con 0x7f30140f6590 2024-02-20T13:51:26.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.993+0000 7f2ffb7fe700 1 -- 172.21.15.192:0/396699628 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f300401d7f0 con 0x7f30140f6590 2024-02-20T13:51:26.980 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:26.980 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:26.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 msgr2=0x7f2ffc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 0x7f2ffc0335a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f30080096a0 tx=0x7f3008006b40).stop 2024-02-20T13:51:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 msgr2=0x7f301406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f30040089f0 tx=0x7f300400b040).stop 2024-02-20T13:51:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 shutdown_connections 2024-02-20T13:51:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f2ffc0310e0 0x7f2ffc0335a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 --2- 172.21.15.192:0/396699628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30140f6590 0x7f301406ba80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 >> 172.21.15.192:0/396699628 conn(0x7f30140f1cf0 msgr2=0x7f30140f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 shutdown_connections 2024-02-20T13:51:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:26.995+0000 7f3019084700 1 -- 172.21.15.192:0/396699628 wait complete. 2024-02-20T13:51:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:27.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:25.426402+0000 mgr.smithi158.gmoobf (mgr.14156) 330 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:27.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:26.598493+0000 mon.smithi158 (mon.0) 309 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:26.994575+0000 mon.smithi158 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.192:0/396699628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:27.083334+0000 mon.smithi158 (mon.0) 311 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:27.084179+0000 mon.smithi158 (mon.0) 312 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-02-20T13:51:27.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:27 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:27.087320+0000 mon.smithi158 (mon.0) 313 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' 2024-02-20T13:51:28.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:27.085215+0000 mgr.smithi158.gmoobf (mgr.14156) 331 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:28.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cephadm 2024-02-20T13:51:27.091421+0000 mgr.smithi158.gmoobf (mgr.14156) 332 : cephadm [INF] Deploying daemon alertmanager.smithi158 on smithi158 2024-02-20T13:51:28.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:28.081447+0000 mon.smithi158 (mon.0) 314 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-20T13:51:28.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:28.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:30.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3359587813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 msgr2=0x7fba9c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:30.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3359587813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c0f58a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fba90009fb0 tx=0x7fba90009b30).stop 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3359587813 shutdown_connections 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3359587813 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c0f58a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3359587813 >> 172.21.15.192:0/3359587813 conn(0x7fba9c0f0be0 msgr2=0x7fba9c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3359587813 shutdown_connections 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3359587813 wait complete. 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 Processor -- start 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.436+0000 7fbaa2de9700 1 -- start start 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa2de9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa2de9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c106cb0 con 0x7fba9c0f5480 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa0b85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa0b85700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:42826/0 (socket says 172.21.15.192:42826) 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa0b85700 1 -- 172.21.15.192:0/3854060742 learned_addr learned my addr 172.21.15.192:0/3854060742 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa0b85700 1 -- 172.21.15.192:0/3854060742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90005040 con 0x7fba9c0f5480 2024-02-20T13:51:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.437+0000 7fbaa0b85700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fba900089f0 tx=0x7fba9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.438+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90009320 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.438+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c106eb0 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.438+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fba90013ec0 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.438+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90014d30 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.438+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c107340 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fba900095f0 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c045bb0 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fba997fa700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 0x7fba8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fba90042390 con 0x7fba9c0f5480 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fba9bfff700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 0x7fba8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.439+0000 7fba9bfff700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 0x7fba8c033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.444+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fba9001d7f0 con 0x7fba9c0f5480 2024-02-20T13:51:30.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:29.085485+0000 mgr.smithi158.gmoobf (mgr.14156) 333 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:30.635 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.649+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba9c1081f0 con 0x7fba9c0f5480 2024-02-20T13:51:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.650+0000 7fba997fa700 1 -- 172.21.15.192:0/3854060742 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba9001d7f0 con 0x7fba9c0f5480 2024-02-20T13:51:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:30.636 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 msgr2=0x7fba8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 0x7fba8c033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).stop 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 msgr2=0x7fba9c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fba900089f0 tx=0x7fba9000b040).stop 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 shutdown_connections 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fba8c031090 0x7fba8c033550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 --2- 172.21.15.192:0/3854060742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9c0f5480 0x7fba9c1067c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 >> 172.21.15.192:0/3854060742 conn(0x7fba9c0f0be0 msgr2=0x7fba9c05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 shutdown_connections 2024-02-20T13:51:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:30.652+0000 7fbaa2de9700 1 -- 172.21.15.192:0/3854060742 wait complete. 2024-02-20T13:51:30.639 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:31.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:31 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:30.651589+0000 mon.smithi158 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.192:0/3854060742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:32.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:32.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:32.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:32 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:31.085888+0000 mgr.smithi158.gmoobf (mgr.14156) 334 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- 172.21.15.192:0/1988301823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 msgr2=0x7f76e40f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/1988301823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e40f5e70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f76d4009fb0 tx=0x7f76d4009b30).stop 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- 172.21.15.192:0/1988301823 shutdown_connections 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/1988301823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e40f5e70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- 172.21.15.192:0/1988301823 >> 172.21.15.192:0/1988301823 conn(0x7f76e40f0360 msgr2=0x7f76e40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- 172.21.15.192:0/1988301823 shutdown_connections 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- 172.21.15.192:0/1988301823 wait complete. 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 Processor -- start 2024-02-20T13:51:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.162+0000 7f76eb8c1700 1 -- start start 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76eb8c1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76eb8c1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4105be0 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76e965d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76e965d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58838/0 (socket says 172.21.15.192:58838) 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76e965d700 1 -- 172.21.15.192:0/2790976424 learned_addr learned my addr 172.21.15.192:0/2790976424 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76e965d700 1 -- 172.21.15.192:0/2790976424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4005040 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76e965d700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f76d40089f0 tx=0x7f76d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76d4009320 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e4105de0 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f76d4013ec0 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.163+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76d4014d30 con 0x7f76e40f5a50 2024-02-20T13:51:34.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.164+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4106270 con 0x7f76e40f5a50 2024-02-20T13:51:34.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.164+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f76d40095f0 con 0x7f76e40f5a50 2024-02-20T13:51:34.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.164+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e4045bb0 con 0x7f76e40f5a50 2024-02-20T13:51:34.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.165+0000 7f76d9ffb700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 0x7f76d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:34.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.165+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f76d4042450 con 0x7f76e40f5a50 2024-02-20T13:51:34.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.165+0000 7f76e8e5c700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 0x7f76d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:34.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.165+0000 7f76e8e5c700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 0x7f76d0033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f76e00096a0 tx=0x7f76e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:34.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.170+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f76d401d7f0 con 0x7f76e40f5a50 2024-02-20T13:51:34.359 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.373+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76e40faf70 con 0x7f76e40f5a50 2024-02-20T13:51:34.360 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.374+0000 7f76d9ffb700 1 -- 172.21.15.192:0/2790976424 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76d401d7f0 con 0x7f76e40f5a50 2024-02-20T13:51:34.360 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:34.360 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:34.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 msgr2=0x7f76d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:34.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 0x7f76d0033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f76e00096a0 tx=0x7f76e0006b40).stop 2024-02-20T13:51:34.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 msgr2=0x7f76e41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:34.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f76d40089f0 tx=0x7f76d400b040).stop 2024-02-20T13:51:34.361 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 shutdown_connections 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f76d0031090 0x7f76d0033550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 --2- 172.21.15.192:0/2790976424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76e40f5a50 0x7f76e41056f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.375+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 >> 172.21.15.192:0/2790976424 conn(0x7f76e40f0360 msgr2=0x7f76e40f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.376+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 shutdown_connections 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:34.376+0000 7f76eb8c1700 1 -- 172.21.15.192:0/2790976424 wait complete. 2024-02-20T13:51:34.362 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:34.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:33.086147+0000 mgr.smithi158.gmoobf (mgr.14156) 335 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:35.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:35 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:34.375326+0000 mon.smithi158 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.192:0/2790976424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:36.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:36.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:36.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:35.086434+0000 mgr.smithi158.gmoobf (mgr.14156) 336 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:37.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 -- 172.21.15.192:0/2264547259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 msgr2=0x7f83b00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:37.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 --2- 172.21.15.192:0/2264547259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b00f69b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f83a0004760 tx=0x7f83a0009ad0).stop 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 -- 172.21.15.192:0/2264547259 shutdown_connections 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 --2- 172.21.15.192:0/2264547259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b00f69b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 -- 172.21.15.192:0/2264547259 >> 172.21.15.192:0/2264547259 conn(0x7f83b00f1cd0 msgr2=0x7f83b00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 -- 172.21.15.192:0/2264547259 shutdown_connections 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.009+0000 7f83b6365700 1 -- 172.21.15.192:0/2264547259 wait complete. 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83b6365700 1 Processor -- start 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83b6365700 1 -- start start 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83b6365700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83b6365700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0106cb0 con 0x7f83b00f6590 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58862/0 (socket says 172.21.15.192:58862) 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.010+0000 7f83affff700 1 -- 172.21.15.192:0/4189559815 learned_addr learned my addr 172.21.15.192:0/4189559815 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83affff700 1 -- 172.21.15.192:0/4189559815 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a0009d70 con 0x7f83b00f6590 2024-02-20T13:51:37.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83affff700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83a0004170 tx=0x7f83a0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a000e040 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b0106eb0 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f83a0008ef0 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a0014c90 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.011+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b0107340 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.012+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f83a0014420 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.012+0000 7f83acff9700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 0x7f8398033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.012+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0045bb0 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.012+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f83a00422e0 con 0x7f83b00f6590 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.012+0000 7f83af7fe700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 0x7f8398033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:37.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.013+0000 7f83af7fe700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 0x7f8398033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f83a40096a0 tx=0x7f83a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:38.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.017+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f83a0026300 con 0x7f83b00f6590 2024-02-20T13:51:38.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.222+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83b01081c0 con 0x7f83b00f6590 2024-02-20T13:51:38.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.222+0000 7f83acff9700 1 -- 172.21.15.192:0/4189559815 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f83a0012b20 con 0x7f83b00f6590 2024-02-20T13:51:38.207 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:38.208 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 msgr2=0x7f8398033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 0x7f8398033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f83a40096a0 tx=0x7f83a4006b40).stop 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 msgr2=0x7f83b01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83a0004170 tx=0x7f83a0009ad0).stop 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 shutdown_connections 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f8398031090 0x7f8398033550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 --2- 172.21.15.192:0/4189559815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83b00f6590 0x7f83b01067c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:38.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 >> 172.21.15.192:0/4189559815 conn(0x7f83b00f1cd0 msgr2=0x7f83b00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:38.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 shutdown_connections 2024-02-20T13:51:38.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:38.224+0000 7f83b6365700 1 -- 172.21.15.192:0/4189559815 wait complete. 2024-02-20T13:51:38.210 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:38.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:37.086710+0000 mgr.smithi158.gmoobf (mgr.14156) 337 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:38.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:38.222930+0000 mon.smithi158 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.192:0/4189559815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:39.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:39.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:40.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:39.086951+0000 mgr.smithi158.gmoobf (mgr.14156) 338 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 -- 172.21.15.192:0/3905927571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 msgr2=0x7f11780f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3905927571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11780f58a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f116c009fb0 tx=0x7f116c009b30).stop 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 -- 172.21.15.192:0/3905927571 shutdown_connections 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3905927571 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11780f58a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 -- 172.21.15.192:0/3905927571 >> 172.21.15.192:0/3905927571 conn(0x7f11780f0be0 msgr2=0x7f11780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 -- 172.21.15.192:0/3905927571 shutdown_connections 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.720+0000 7f117d8a1700 1 -- 172.21.15.192:0/3905927571 wait complete. 2024-02-20T13:51:41.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f117d8a1700 1 Processor -- start 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f117d8a1700 1 -- start start 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f117d8a1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f117d8a1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178108ed0 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f1176ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f1176ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:58884/0 (socket says 172.21.15.192:58884) 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f1176ffd700 1 -- 172.21.15.192:0/3674507611 learned_addr learned my addr 172.21.15.192:0/3674507611 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.721+0000 7f1176ffd700 1 -- 172.21.15.192:0/3674507611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c005040 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f1176ffd700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f116c0089f0 tx=0x7f116c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c009320 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f116c013ec0 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11781090d0 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c014d30 con 0x7f11780f5480 2024-02-20T13:51:41.719 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.722+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1178109560 con 0x7f11780f5480 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.723+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f116c014420 con 0x7f11780f5480 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.723+0000 7f115f7fe700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 0x7f11600335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.723+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f116c042410 con 0x7f11780f5480 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.723+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1178045bb0 con 0x7f11780f5480 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.723+0000 7f11767fc700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 0x7f11600335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.724+0000 7f11767fc700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 0x7f11600335a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f11680096a0 tx=0x7f1168006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:41.720 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.728+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f116c014750 con 0x7f11780f5480 2024-02-20T13:51:41.917 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.932+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11780fb060 con 0x7f11780f5480 2024-02-20T13:51:41.918 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.933+0000 7f115f7fe700 1 -- 172.21.15.192:0/3674507611 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f116c01d7f0 con 0x7f11780f5480 2024-02-20T13:51:41.918 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:41.918 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:41.919 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 msgr2=0x7f11600335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 0x7f11600335a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f11680096a0 tx=0x7f1168006b40).stop 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 msgr2=0x7f11781089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f116c0089f0 tx=0x7f116c00b040).stop 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 shutdown_connections 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f11600310e0 0x7f11600335a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 --2- 172.21.15.192:0/3674507611 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11780f5480 0x7f11781089e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 >> 172.21.15.192:0/3674507611 conn(0x7f11780f0be0 msgr2=0x7f117805ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 shutdown_connections 2024-02-20T13:51:41.920 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:41.935+0000 7f117d8a1700 1 -- 172.21.15.192:0/3674507611 wait complete. 2024-02-20T13:51:41.921 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:42.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:41.087271+0000 mgr.smithi158.gmoobf (mgr.14156) 339 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:42.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:42 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:41.933458+0000 mon.smithi158 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.192:0/3674507611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:43.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:43.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:44.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:44 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:43.087529+0000 mgr.smithi158.gmoobf (mgr.14156) 340 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:45.469 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 -- 172.21.15.192:0/4133922997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 msgr2=0x7f6c980f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/4133922997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c980f91c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6c88009fb0 tx=0x7f6c88009b30).stop 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 -- 172.21.15.192:0/4133922997 shutdown_connections 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/4133922997 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c980f91c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 -- 172.21.15.192:0/4133922997 >> 172.21.15.192:0/4133922997 conn(0x7f6c980f0be0 msgr2=0x7f6c980f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 -- 172.21.15.192:0/4133922997 shutdown_connections 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.483+0000 7f6c9e11a700 1 -- 172.21.15.192:0/4133922997 wait complete. 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c9e11a700 1 Processor -- start 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c9e11a700 1 -- start start 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c9e11a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c9e11a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98106c70 con 0x7f6c98060300 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c977fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c977fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51846/0 (socket says 172.21.15.192:51846) 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.484+0000 7f6c977fe700 1 -- 172.21.15.192:0/3840570764 learned_addr learned my addr 172.21.15.192:0/3840570764 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c977fe700 1 -- 172.21.15.192:0/3840570764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6c98060300 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c977fe700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6c88009f80 tx=0x7f6c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c88009320 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c98106e70 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6c88013680 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8801eda0 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.485+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c98107300 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f6c8801e420 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c8ffff700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 0x7f6c84033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6c880423c0 con 0x7f6c98060300 2024-02-20T13:51:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c98045bb0 con 0x7f6c98060300 2024-02-20T13:51:45.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c96ffd700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 0x7f6c84033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:45.473 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.486+0000 7f6c96ffd700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 0x7f6c84033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:45.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.491+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6c8801e700 con 0x7f6c98060300 2024-02-20T13:51:45.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.697+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c98108020 con 0x7f6c98060300 2024-02-20T13:51:45.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.698+0000 7f6c8ffff700 1 -- 172.21.15.192:0/3840570764 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c8801cb10 con 0x7f6c98060300 2024-02-20T13:51:45.683 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:45.683 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 msgr2=0x7f6c84033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 0x7f6c84033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).stop 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 msgr2=0x7f6c98106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6c88009f80 tx=0x7f6c8800b040).stop 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 shutdown_connections 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f6c84031090 0x7f6c84033550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 --2- 172.21.15.192:0/3840570764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c98060300 0x7f6c98106780 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.699+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 >> 172.21.15.192:0/3840570764 conn(0x7f6c980f0be0 msgr2=0x7f6c980f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:45.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.700+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 shutdown_connections 2024-02-20T13:51:45.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:45.700+0000 7f6c9e11a700 1 -- 172.21.15.192:0/3840570764 wait complete. 2024-02-20T13:51:45.686 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:46.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:45.087840+0000 mgr.smithi158.gmoobf (mgr.14156) 341 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:46.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:46 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:45.698015+0000 mon.smithi158 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.192:0/3840570764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:47.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:47.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:48.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:48 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:47.088171+0000 mgr.smithi158.gmoobf (mgr.14156) 342 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 -- 172.21.15.192:0/2781463375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 msgr2=0x7fabb80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/2781463375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb80f58a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fabac009fb0 tx=0x7fabac009b30).stop 2024-02-20T13:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 -- 172.21.15.192:0/2781463375 shutdown_connections 2024-02-20T13:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/2781463375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb80f58a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 -- 172.21.15.192:0/2781463375 >> 172.21.15.192:0/2781463375 conn(0x7fabb80f0be0 msgr2=0x7fabb80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 -- 172.21.15.192:0/2781463375 shutdown_connections 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.285+0000 7fabbd9ba700 1 -- 172.21.15.192:0/2781463375 wait complete. 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabbd9ba700 1 Processor -- start 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabbd9ba700 1 -- start start 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabbd9ba700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabbd9ba700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8060360 con 0x7fabb80f5480 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabb6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabb6ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:51864/0 (socket says 172.21.15.192:51864) 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.286+0000 7fabb6ffd700 1 -- 172.21.15.192:0/229625271 learned_addr learned my addr 172.21.15.192:0/229625271 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fabb6ffd700 1 -- 172.21.15.192:0/229625271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac005040 con 0x7fabb80f5480 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fabb6ffd700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fabac0089f0 tx=0x7fabac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabac009320 con 0x7fabb80f5480 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb8060560 con 0x7fabb80f5480 2024-02-20T13:51:49.274 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb8109be0 con 0x7fabb80f5480 2024-02-20T13:51:49.275 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fabac013ec0 con 0x7fabb80f5480 2024-02-20T13:51:49.275 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.287+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabac014d00 con 0x7fabb80f5480 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.288+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fabac014420 con 0x7fabb80f5480 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.288+0000 7fab9f7fe700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 0x7faba0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.288+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb8045bb0 con 0x7fabb80f5480 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.288+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fabac042470 con 0x7fabb80f5480 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.288+0000 7fabb67fc700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 0x7faba0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:49.276 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.289+0000 7fabb67fc700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 0x7faba0033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7faba80096a0 tx=0x7faba8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:49.278 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.293+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fabac01c320 con 0x7fabb80f5480 2024-02-20T13:51:49.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.498+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabb810a400 con 0x7fabb80f5480 2024-02-20T13:51:49.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.499+0000 7fab9f7fe700 1 -- 172.21.15.192:0/229625271 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fabac01d380 con 0x7fabb80f5480 2024-02-20T13:51:49.485 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:49.485 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:49.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 msgr2=0x7faba0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 0x7faba0033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7faba80096a0 tx=0x7faba8006b40).stop 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 msgr2=0x7fabb81094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fabac0089f0 tx=0x7fabac00b040).stop 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 shutdown_connections 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7faba0031090 0x7faba0033550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 --2- 172.21.15.192:0/229625271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabb80f5480 0x7fabb81094b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 >> 172.21.15.192:0/229625271 conn(0x7fabb80f0be0 msgr2=0x7fabb80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 shutdown_connections 2024-02-20T13:51:49.487 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:49.501+0000 7fabbd9ba700 1 -- 172.21.15.192:0/229625271 wait complete. 2024-02-20T13:51:49.488 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:50.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:49.088558+0000 mgr.smithi158.gmoobf (mgr.14156) 343 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:50.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:50 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:49.500461+0000 mon.smithi158 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.192:0/229625271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:51.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:51.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:52.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:52 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:51.088839+0000 mgr.smithi158.gmoobf (mgr.14156) 344 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 -- 172.21.15.192:0/2525098896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 msgr2=0x7fa9cc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/2525098896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc0f69b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa9c0004760 tx=0x7fa9c0009ad0).stop 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 -- 172.21.15.192:0/2525098896 shutdown_connections 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/2525098896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc0f69b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 -- 172.21.15.192:0/2525098896 >> 172.21.15.192:0/2525098896 conn(0x7fa9cc0f1cd0 msgr2=0x7fa9cc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 -- 172.21.15.192:0/2525098896 shutdown_connections 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.335+0000 7fa9d0a18700 1 -- 172.21.15.192:0/2525098896 wait complete. 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9d0a18700 1 Processor -- start 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9d0a18700 1 -- start start 2024-02-20T13:51:53.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9d0a18700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9d0a18700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc108f00 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:49678/0 (socket says 172.21.15.192:49678) 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.336+0000 7fa9ca59c700 1 -- 172.21.15.192:0/661270761 learned_addr learned my addr 172.21.15.192:0/661270761 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9ca59c700 1 -- 172.21.15.192:0/661270761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0009d70 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9ca59c700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa9c0004170 tx=0x7fa9c0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c000e040 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc109100 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa9c0008ef0 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c0014c90 con 0x7fa9cc0f6590 2024-02-20T13:51:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.337+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc109590 con 0x7fa9cc0f6590 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.338+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa9c0014420 con 0x7fa9cc0f6590 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.338+0000 7fa9baffd700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 0x7fa9b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.338+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc045bb0 con 0x7fa9cc0f6590 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.338+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa9c00422d0 con 0x7fa9cc0f6590 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.338+0000 7fa9c9d9b700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 0x7fa9b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.339+0000 7fa9c9d9b700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 0x7fa9b4033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa9bc0096a0 tx=0x7fa9bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:53.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.343+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa9c0012d00 con 0x7fa9cc0f6590 2024-02-20T13:51:53.536 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.550+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9cc10a3c0 con 0x7fa9cc0f6590 2024-02-20T13:51:53.536 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.551+0000 7fa9baffd700 1 -- 172.21.15.192:0/661270761 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9c0012b20 con 0x7fa9cc0f6590 2024-02-20T13:51:53.537 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:53.537 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:53.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 msgr2=0x7fa9b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 0x7fa9b4033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa9bc0096a0 tx=0x7fa9bc006b40).stop 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 msgr2=0x7fa9cc108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa9c0004170 tx=0x7fa9c0009ad0).stop 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 shutdown_connections 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa9b4031090 0x7fa9b4033550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 --2- 172.21.15.192:0/661270761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9cc0f6590 0x7fa9cc108a10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 >> 172.21.15.192:0/661270761 conn(0x7fa9cc0f1cd0 msgr2=0x7fa9cc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 shutdown_connections 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:53.552+0000 7fa9d0a18700 1 -- 172.21.15.192:0/661270761 wait complete. 2024-02-20T13:51:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:54.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:54 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:53.089157+0000 mgr.smithi158.gmoobf (mgr.14156) 345 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:54.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:54 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:53.552235+0000 mon.smithi158 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.192:0/661270761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:55.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:55.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:51:56.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:55.089487+0000 mgr.smithi158.gmoobf (mgr.14156) 346 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:56.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:55.470657+0000 mon.smithi158 (mon.0) 322 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/trash_purge_schedule"}]: dispatch 2024-02-20T13:51:56.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:56 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:55.470866+0000 mon.smithi158 (mon.0) 323 : audit [INF] from='mgr.14156 172.21.15.158:0/2483527735' entity='mgr.smithi158.gmoobf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi158.gmoobf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:51:57.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 -- 172.21.15.192:0/2798190425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 msgr2=0x7f56980f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:57.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 --2- 172.21.15.192:0/2798190425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f56980f69b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f568c004760 tx=0x7f568c009ad0).stop 2024-02-20T13:51:57.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 -- 172.21.15.192:0/2798190425 shutdown_connections 2024-02-20T13:51:57.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 --2- 172.21.15.192:0/2798190425 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f56980f69b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:57.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 -- 172.21.15.192:0/2798190425 >> 172.21.15.192:0/2798190425 conn(0x7f56980f1cd0 msgr2=0x7f56980f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 -- 172.21.15.192:0/2798190425 shutdown_connections 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.107+0000 7f569e874700 1 -- 172.21.15.192:0/2798190425 wait complete. 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f569e874700 1 Processor -- start 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f569e874700 1 -- start start 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f569e874700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f569e874700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5698069f00 con 0x7f56980f6590 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f5697fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f5697fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:49704/0 (socket says 172.21.15.192:49704) 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.108+0000 7f5697fff700 1 -- 172.21.15.192:0/668086738 learned_addr learned my addr 172.21.15.192:0/668086738 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f5697fff700 1 -- 172.21.15.192:0/668086738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c009d70 con 0x7f56980f6590 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f5697fff700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f568c013f10 tx=0x7f568c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c00e040 con 0x7f56980f6590 2024-02-20T13:51:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569806a0a0 con 0x7f56980f6590 2024-02-20T13:51:57.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f568c007df0 con 0x7f56980f6590 2024-02-20T13:51:57.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c014d60 con 0x7f56980f6590 2024-02-20T13:51:57.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.109+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569806a530 con 0x7f56980f6590 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.110+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f568c014520 con 0x7f56980f6590 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.110+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56980591e0 con 0x7f56980f6590 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.110+0000 7f5694ff9700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.110+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f568c042550 con 0x7f56980f6590 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.110+0000 7f56977fe700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:51:57.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.111+0000 7f56977fe700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 0x7f5680033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f56880096a0 tx=0x7f5688006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:51:57.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.115+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f568c007960 con 0x7f56980f6590 2024-02-20T13:51:57.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.323+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56980fc080 con 0x7f56980f6590 2024-02-20T13:51:57.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.323+0000 7f5694ff9700 1 -- 172.21.15.192:0/668086738 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f568c0262f0 con 0x7f56980f6590 2024-02-20T13:51:57.309 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:51:57.309 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 msgr2=0x7f5680033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 0x7f5680033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f56880096a0 tx=0x7f5688006b40).stop 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 msgr2=0x7f569806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f568c013f10 tx=0x7f568c009ad0).stop 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 shutdown_connections 2024-02-20T13:51:57.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:57.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 --2- 172.21.15.192:0/668086738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f56980f6590 0x7f569806ba80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:51:57.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 >> 172.21.15.192:0/668086738 conn(0x7f56980f1cd0 msgr2=0x7f56980f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:51:57.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 shutdown_connections 2024-02-20T13:51:57.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:51:57.325+0000 7f569e874700 1 -- 172.21.15.192:0/668086738 wait complete. 2024-02-20T13:51:57.312 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:51:58.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:57.089914+0000 mgr.smithi158.gmoobf (mgr.14156) 347 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:51:58.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:51:58 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:51:57.324723+0000 mon.smithi158 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.192:0/668086738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:51:59.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:51:59.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:00.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:00 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:51:59.090268+0000 mgr.smithi158.gmoobf (mgr.14156) 348 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:01.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 -- 172.21.15.192:0/1005771118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 msgr2=0x7f54940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:01.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 --2- 172.21.15.192:0/1005771118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f54940f58a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f5488009fb0 tx=0x7f5488009b30).stop 2024-02-20T13:52:01.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 -- 172.21.15.192:0/1005771118 shutdown_connections 2024-02-20T13:52:01.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 --2- 172.21.15.192:0/1005771118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f54940f58a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:01.017 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 -- 172.21.15.192:0/1005771118 >> 172.21.15.192:0/1005771118 conn(0x7f54940f0be0 msgr2=0x7f54940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:01.017 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 -- 172.21.15.192:0/1005771118 shutdown_connections 2024-02-20T13:52:01.017 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.030+0000 7f54998c7700 1 -- 172.21.15.192:0/1005771118 wait complete. 2024-02-20T13:52:01.018 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f54998c7700 1 Processor -- start 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f54998c7700 1 -- start start 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f54998c7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f54998c7700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494060300 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f5492ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f5492ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:49718/0 (socket says 172.21.15.192:49718) 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.031+0000 7f5492ffd700 1 -- 172.21.15.192:0/1869171669 learned_addr learned my addr 172.21.15.192:0/1869171669 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f5492ffd700 1 -- 172.21.15.192:0/1869171669 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5488005040 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f5492ffd700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f54880089f0 tx=0x7f548800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5488009320 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5488013ec0 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5494060500 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5488014d30 con 0x7f54940f5480 2024-02-20T13:52:01.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.032+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5494109bc0 con 0x7f54940f5480 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f54880095f0 con 0x7f54940f5480 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f547b7fe700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 0x7f547c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5488042450 con 0x7f54940f5480 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54940591e0 con 0x7f54940f5480 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f54927fc700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 0x7f547c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:01.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.033+0000 7f54927fc700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 0x7f547c033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f54840096a0 tx=0x7f5484006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:01.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.038+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f548801d7f0 con 0x7f54940f5480 2024-02-20T13:52:01.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.247+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5494045bb0 con 0x7f54940f5480 2024-02-20T13:52:01.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.248+0000 7f547b7fe700 1 -- 172.21.15.192:0/1869171669 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f548801d7f0 con 0x7f54940f5480 2024-02-20T13:52:01.234 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:01.234 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:01.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 msgr2=0x7f547c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:01.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 0x7f547c033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f54840096a0 tx=0x7f5484006b40).stop 2024-02-20T13:52:01.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 msgr2=0x7f5494109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f54880089f0 tx=0x7f548800b040).stop 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 shutdown_connections 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f547c031090 0x7f547c033550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 --2- 172.21.15.192:0/1869171669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54940f5480 0x7f5494109510 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 >> 172.21.15.192:0/1869171669 conn(0x7f54940f0be0 msgr2=0x7f54940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 shutdown_connections 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:01.250+0000 7f54998c7700 1 -- 172.21.15.192:0/1869171669 wait complete. 2024-02-20T13:52:01.237 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:01.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:01 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:01.249573+0000 mon.smithi158 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.192:0/1869171669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:02.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:02 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:01.090630+0000 mgr.smithi158.gmoobf (mgr.14156) 349 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:02.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:02.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:04.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:04 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:03.091037+0000 mgr.smithi158.gmoobf (mgr.14156) 350 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- 172.21.15.192:0/1225527459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 msgr2=0x7fa6480f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/1225527459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6480f58a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa644009fb0 tx=0x7fa644009b30).stop 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- 172.21.15.192:0/1225527459 shutdown_connections 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/1225527459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6480f58a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- 172.21.15.192:0/1225527459 >> 172.21.15.192:0/1225527459 conn(0x7fa6480f0be0 msgr2=0x7fa6480f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- 172.21.15.192:0/1225527459 shutdown_connections 2024-02-20T13:52:04.680 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- 172.21.15.192:0/1225527459 wait complete. 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 Processor -- start 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- start start 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.690+0000 7fa64fe9a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648105be0 con 0x7fa6480f5480 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64dc36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64dc36700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55590/0 (socket says 172.21.15.192:55590) 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64dc36700 1 -- 172.21.15.192:0/2587496239 learned_addr learned my addr 172.21.15.192:0/2587496239 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64dc36700 1 -- 172.21.15.192:0/2587496239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa644005040 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64dc36700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa6440089f0 tx=0x7fa64400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa644009320 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa648105de0 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa644013ec0 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa644014d30 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.691+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648106270 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.692+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa6440263a0 con 0x7fa6480f5480 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.692+0000 7fa63e7fc700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 0x7fa634033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:04.682 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.692+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa6440424a0 con 0x7fa6480f5480 2024-02-20T13:52:04.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.692+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6480591e0 con 0x7fa6480f5480 2024-02-20T13:52:04.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.692+0000 7fa64d435700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 0x7fa634033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:04.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.693+0000 7fa64d435700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 0x7fa634033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:04.683 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.697+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa644012920 con 0x7fa6480f5480 2024-02-20T13:52:04.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.904+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6480faf70 con 0x7fa6480f5480 2024-02-20T13:52:04.890 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.905+0000 7fa63e7fc700 1 -- 172.21.15.192:0/2587496239 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa64402b020 con 0x7fa6480f5480 2024-02-20T13:52:04.890 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:04.890 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 msgr2=0x7fa634033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 0x7fa634033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).stop 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 msgr2=0x7fa6481056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa6440089f0 tx=0x7fa64400b040).stop 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 shutdown_connections 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa634031090 0x7fa634033550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:04.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 --2- 172.21.15.192:0/2587496239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6480f5480 0x7fa6481056f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:04.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 >> 172.21.15.192:0/2587496239 conn(0x7fa6480f0be0 msgr2=0x7fa6480f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:04.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 shutdown_connections 2024-02-20T13:52:04.893 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:04.906+0000 7fa64fe9a700 1 -- 172.21.15.192:0/2587496239 wait complete. 2024-02-20T13:52:04.893 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:05.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:05 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:04.906077+0000 mon.smithi158 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.192:0/2587496239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:06.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:06 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:05.091393+0000 mgr.smithi158.gmoobf (mgr.14156) 351 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:06.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:06.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:07.781 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:07 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:07.091751+0000 mgr.smithi158.gmoobf (mgr.14156) 352 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- 172.21.15.192:0/2708917846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 msgr2=0x7fc62c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 --2- 172.21.15.192:0/2708917846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c0f58a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fc61c009fb0 tx=0x7fc61c009b30).stop 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- 172.21.15.192:0/2708917846 shutdown_connections 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 --2- 172.21.15.192:0/2708917846 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c0f58a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- 172.21.15.192:0/2708917846 >> 172.21.15.192:0/2708917846 conn(0x7fc62c0f0be0 msgr2=0x7fc62c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- 172.21.15.192:0/2708917846 shutdown_connections 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- 172.21.15.192:0/2708917846 wait complete. 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 Processor -- start 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- start start 2024-02-20T13:52:08.645 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.658+0000 7fc634553700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c060360 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc6322ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc6322ef700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:55600/0 (socket says 172.21.15.192:55600) 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc6322ef700 1 -- 172.21.15.192:0/3301687704 learned_addr learned my addr 172.21.15.192:0/3301687704 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc6322ef700 1 -- 172.21.15.192:0/3301687704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c005040 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc6322ef700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fc61c0089f0 tx=0x7fc61c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc61c009320 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62c060560 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc61c013ec0 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc61c014d30 con 0x7fc62c0f5480 2024-02-20T13:52:08.646 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.659+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62c109be0 con 0x7fc62c0f5480 2024-02-20T13:52:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.660+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fc61c014420 con 0x7fc62c0f5480 2024-02-20T13:52:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.660+0000 7fc622ffd700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 0x7fc618033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.660+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62c045bb0 con 0x7fc62c0f5480 2024-02-20T13:52:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.660+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc61c042390 con 0x7fc62c0f5480 2024-02-20T13:52:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.661+0000 7fc631aee700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 0x7fc618033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:08.648 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.661+0000 7fc631aee700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 0x7fc618033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc6280096a0 tx=0x7fc628006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:08.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.665+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc61c01d7f0 con 0x7fc62c0f5480 2024-02-20T13:52:08.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.869+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc62c10a3d0 con 0x7fc62c0f5480 2024-02-20T13:52:08.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.869+0000 7fc622ffd700 1 -- 172.21.15.192:0/3301687704 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc61c02c090 con 0x7fc62c0f5480 2024-02-20T13:52:08.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:08.855 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 msgr2=0x7fc618033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 0x7fc618033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc6280096a0 tx=0x7fc628006b40).stop 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 msgr2=0x7fc62c1094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fc61c0089f0 tx=0x7fc61c00b040).stop 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 shutdown_connections 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fc618031090 0x7fc618033550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 --2- 172.21.15.192:0/3301687704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc62c0f5480 0x7fc62c1094b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 >> 172.21.15.192:0/3301687704 conn(0x7fc62c0f0be0 msgr2=0x7fc62c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 shutdown_connections 2024-02-20T13:52:08.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:08.871+0000 7fc634553700 1 -- 172.21.15.192:0/3301687704 wait complete. 2024-02-20T13:52:08.858 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:09.281 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:08 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:08.871054+0000 mon.smithi158 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.192:0/3301687704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:10.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:10 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:09.092078+0000 mgr.smithi158.gmoobf (mgr.14156) 353 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:10.542 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:10.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:12.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1789101565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 msgr2=0x7fa2f40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:12.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1789101565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f40f69b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa2f0004760 tx=0x7fa2f0009ad0).stop 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1789101565 shutdown_connections 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1789101565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f40f69b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1789101565 >> 172.21.15.192:0/1789101565 conn(0x7fa2f40f1cd0 msgr2=0x7fa2f40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1789101565 shutdown_connections 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1789101565 wait complete. 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 Processor -- start 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.419+0000 7fa2fbc47700 1 -- start start 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2fbc47700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2fbc47700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4108ea0 con 0x7fa2f40f6590 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2f99e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2f99e3700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39512/0 (socket says 172.21.15.192:39512) 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2f99e3700 1 -- 172.21.15.192:0/1896203734 learned_addr learned my addr 172.21.15.192:0/1896203734 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2f99e3700 1 -- 172.21.15.192:0/1896203734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f0009d70 con 0x7fa2f40f6590 2024-02-20T13:52:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2f99e3700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa2f0004170 tx=0x7fa2f0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:12.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2f000e040 con 0x7fa2f40f6590 2024-02-20T13:52:12.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f41090a0 con 0x7fa2f40f6590 2024-02-20T13:52:12.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2f0008ef0 con 0x7fa2f40f6590 2024-02-20T13:52:12.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2f0014c90 con 0x7fa2f40f6590 2024-02-20T13:52:12.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.420+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4109530 con 0x7fa2f40f6590 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.421+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fa2f0014420 con 0x7fa2f40f6590 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.421+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f4045bb0 con 0x7fa2f40f6590 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.421+0000 7fa2ea7fc700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 0x7fa2e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.421+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2f00422d0 con 0x7fa2f40f6590 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.422+0000 7fa2f91e2700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 0x7fa2e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:12.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.422+0000 7fa2f91e2700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 0x7fa2e0033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:12.412 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.426+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2f0012d00 con 0x7fa2f40f6590 2024-02-20T13:52:12.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:12 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:11.092312+0000 mgr.smithi158.gmoobf (mgr.14156) 354 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:12.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.631+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2f406b140 con 0x7fa2f40f6590 2024-02-20T13:52:12.617 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.632+0000 7fa2ea7fc700 1 -- 172.21.15.192:0/1896203734 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa2f0012b20 con 0x7fa2f40f6590 2024-02-20T13:52:12.618 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:12.618 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 msgr2=0x7fa2e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 0x7fa2e0033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).stop 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 msgr2=0x7fa2f41089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa2f0004170 tx=0x7fa2f0009ad0).stop 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 shutdown_connections 2024-02-20T13:52:12.619 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fa2e0031090 0x7fa2e0033550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:12.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 --2- 172.21.15.192:0/1896203734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2f40f6590 0x7fa2f41089b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:12.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 >> 172.21.15.192:0/1896203734 conn(0x7fa2f40f1cd0 msgr2=0x7fa2f40f2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:12.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 shutdown_connections 2024-02-20T13:52:12.620 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:12.633+0000 7fa2fbc47700 1 -- 172.21.15.192:0/1896203734 wait complete. 2024-02-20T13:52:12.620 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:13.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:13 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:12.633276+0000 mon.smithi158 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.192:0/1896203734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:14.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:14.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:14.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:14 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:13.092667+0000 mgr.smithi158.gmoobf (mgr.14156) 355 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:16.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- 172.21.15.192:0/16860076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 msgr2=0x7f55500f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:16.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 --2- 172.21.15.192:0/16860076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f55500f56f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f5540009fb0 tx=0x7f5540009b30).stop 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- 172.21.15.192:0/16860076 shutdown_connections 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 --2- 172.21.15.192:0/16860076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f55500f56f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- 172.21.15.192:0/16860076 >> 172.21.15.192:0/16860076 conn(0x7f55500f0a30 msgr2=0x7f55500f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- 172.21.15.192:0/16860076 shutdown_connections 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- 172.21.15.192:0/16860076 wait complete. 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 Processor -- start 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- start start 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.321+0000 7f5557560700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550060320 con 0x7f55500f52d0 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f555655e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:16.309 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f555655e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39528/0 (socket says 172.21.15.192:39528) 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f555655e700 1 -- 172.21.15.192:0/864592867 learned_addr learned my addr 172.21.15.192:0/864592867 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f555655e700 1 -- 172.21.15.192:0/864592867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5540005040 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f555655e700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f55400089f0 tx=0x7f554000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5540009320 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5550060520 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5540013ec0 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5540014d30 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.322+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55501099c0 con 0x7f55500f52d0 2024-02-20T13:52:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.323+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f55400263a0 con 0x7f55500f52d0 2024-02-20T13:52:16.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.323+0000 7f5546ffd700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 0x7f553c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:16.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.323+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f55400423e0 con 0x7f55500f52d0 2024-02-20T13:52:16.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.323+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55500591e0 con 0x7f55500f52d0 2024-02-20T13:52:16.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.324+0000 7f5555d5d700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 0x7f553c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:16.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.324+0000 7f5555d5d700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 0x7f553c033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f554c0096a0 tx=0x7f554c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:16.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.328+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f55400266d0 con 0x7f55500f52d0 2024-02-20T13:52:16.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.533+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5550045bb0 con 0x7f55500f52d0 2024-02-20T13:52:16.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.534+0000 7f5546ffd700 1 -- 172.21.15.192:0/864592867 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5540012330 con 0x7f55500f52d0 2024-02-20T13:52:16.520 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:16.520 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 msgr2=0x7f553c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 0x7f553c033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f554c0096a0 tx=0x7f554c006b40).stop 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 msgr2=0x7f5550109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f55400089f0 tx=0x7f554000b040).stop 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 shutdown_connections 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f553c031090 0x7f553c033550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 --2- 172.21.15.192:0/864592867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55500f52d0 0x7f5550109310 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 >> 172.21.15.192:0/864592867 conn(0x7f55500f0a30 msgr2=0x7f55500f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 shutdown_connections 2024-02-20T13:52:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:16.536+0000 7f5557560700 1 -- 172.21.15.192:0/864592867 wait complete. 2024-02-20T13:52:16.523 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:16.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:16 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:15.093037+0000 mgr.smithi158.gmoobf (mgr.14156) 356 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:17.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:17 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:16.535570+0000 mon.smithi158 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.192:0/864592867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:18.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:18.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:18.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:18 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:17.093435+0000 mgr.smithi158.gmoobf (mgr.14156) 357 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- 172.21.15.192:0/4148098003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 msgr2=0x7f926c0f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 --2- 172.21.15.192:0/4148098003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c0f9010 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f9260009fb0 tx=0x7f9260009b30).stop 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- 172.21.15.192:0/4148098003 shutdown_connections 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 --2- 172.21.15.192:0/4148098003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c0f9010 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- 172.21.15.192:0/4148098003 >> 172.21.15.192:0/4148098003 conn(0x7f926c0f0a30 msgr2=0x7f926c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- 172.21.15.192:0/4148098003 shutdown_connections 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- 172.21.15.192:0/4148098003 wait complete. 2024-02-20T13:52:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 Processor -- start 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- start start 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.993+0000 7f926bfff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c106b10 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926affd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:39558/0 (socket says 172.21.15.192:39558) 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926affd700 1 -- 172.21.15.192:0/1641003184 learned_addr learned my addr 172.21.15.192:0/1641003184 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926affd700 1 -- 172.21.15.192:0/1641003184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9260005040 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926affd700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f9260009f80 tx=0x7f926000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9260009320 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c106d10 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9260013680 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f926001eda0 con 0x7f926c060300 2024-02-20T13:52:19.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.994+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c1071a0 con 0x7f926c060300 2024-02-20T13:52:19.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.995+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f926001e420 con 0x7f926c060300 2024-02-20T13:52:19.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.995+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c045bb0 con 0x7f926c060300 2024-02-20T13:52:19.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.995+0000 7f92537fe700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 0x7f9254033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:19.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.995+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9260042420 con 0x7f926c060300 2024-02-20T13:52:19.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.996+0000 7f926a7fc700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 0x7f9254033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:19.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:19.996+0000 7f926a7fc700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 0x7f9254033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f925c0096a0 tx=0x7f925c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:19.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.000+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f926001d380 con 0x7f926c060300 2024-02-20T13:52:20.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.207+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f926c107ff0 con 0x7f926c060300 2024-02-20T13:52:20.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.208+0000 7f92537fe700 1 -- 172.21.15.192:0/1641003184 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f926002c2d0 con 0x7f926c060300 2024-02-20T13:52:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:20.194 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:20.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.209+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 msgr2=0x7f9254033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:20.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.209+0000 7f926bfff700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 0x7f9254033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f925c0096a0 tx=0x7f925c006b40).stop 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.209+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 msgr2=0x7f926c106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.209+0000 7f926bfff700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f9260009f80 tx=0x7f926000b040).stop 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 shutdown_connections 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f9254031090 0x7f9254033550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 --2- 172.21.15.192:0/1641003184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f926c060300 0x7f926c106620 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 >> 172.21.15.192:0/1641003184 conn(0x7f926c0f0a30 msgr2=0x7f926c0f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 shutdown_connections 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:20.210+0000 7f926bfff700 1 -- 172.21.15.192:0/1641003184 wait complete. 2024-02-20T13:52:20.196 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:20.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:19.093851+0000 mgr.smithi158.gmoobf (mgr.14156) 358 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:20.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:20 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:20.209571+0000 mon.smithi158 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.192:0/1641003184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:21.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:21.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:22.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:22 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:21.094246+0000 mgr.smithi158.gmoobf (mgr.14156) 359 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 -- 172.21.15.192:0/2698848694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 msgr2=0x7fbce80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 --2- 172.21.15.192:0/2698848694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce80f69b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fbcdc004760 tx=0x7fbcdc009ad0).stop 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 -- 172.21.15.192:0/2698848694 shutdown_connections 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 --2- 172.21.15.192:0/2698848694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce80f69b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 -- 172.21.15.192:0/2698848694 >> 172.21.15.192:0/2698848694 conn(0x7fbce80f1cd0 msgr2=0x7fbce80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 -- 172.21.15.192:0/2698848694 shutdown_connections 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.878+0000 7fbceef70700 1 -- 172.21.15.192:0/2698848694 wait complete. 2024-02-20T13:52:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbceef70700 1 Processor -- start 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbceef70700 1 -- start start 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbceef70700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbceef70700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8069f00 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbcecd0c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbcecd0c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43222/0 (socket says 172.21.15.192:43222) 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.879+0000 7fbcecd0c700 1 -- 172.21.15.192:0/1447094950 learned_addr learned my addr 172.21.15.192:0/1447094950 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbcecd0c700 1 -- 172.21.15.192:0/1447094950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc009d70 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbcecd0c700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fbcdc004170 tx=0x7fbcdc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcdc00e040 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce806a0a0 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbcdc008ef0 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcdc014c90 con 0x7fbce80f6590 2024-02-20T13:52:23.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.880+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce806a530 con 0x7fbce80f6590 2024-02-20T13:52:23.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.881+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fbcdc0283f0 con 0x7fbce80f6590 2024-02-20T13:52:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.881+0000 7fbce57fa700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 0x7fbcd8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.881+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbcdc0422b0 con 0x7fbce80f6590 2024-02-20T13:52:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.881+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce80591e0 con 0x7fbce80f6590 2024-02-20T13:52:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.881+0000 7fbce7fff700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 0x7fbcd8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.882+0000 7fbce7fff700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 0x7fbcd8033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fbcd40096a0 tx=0x7fbcd4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:23.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:23.886+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbcdc0268f0 con 0x7fbce80f6590 2024-02-20T13:52:24.078 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.093+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbce80008d0 con 0x7fbce80f6590 2024-02-20T13:52:24.079 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.093+0000 7fbce57fa700 1 -- 172.21.15.192:0/1447094950 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcdc026300 con 0x7fbce80f6590 2024-02-20T13:52:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:24.079 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:24.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 msgr2=0x7fbcd8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:24.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 0x7fbcd8033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fbcd40096a0 tx=0x7fbcd4006b40).stop 2024-02-20T13:52:24.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 msgr2=0x7fbce806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fbcdc004170 tx=0x7fbcdc009ad0).stop 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 shutdown_connections 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fbcd8031090 0x7fbcd8033550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 --2- 172.21.15.192:0/1447094950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbce80f6590 0x7fbce806ba80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 >> 172.21.15.192:0/1447094950 conn(0x7fbce80f1cd0 msgr2=0x7fbce80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 shutdown_connections 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:24.095+0000 7fbceef70700 1 -- 172.21.15.192:0/1447094950 wait complete. 2024-02-20T13:52:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:24.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:23.094656+0000 mgr.smithi158.gmoobf (mgr.14156) 360 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:24.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:24 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:24.094972+0000 mon.smithi158 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.192:0/1447094950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:25.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:25.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:26.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:26 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:25.095069+0000 mgr.smithi158.gmoobf (mgr.14156) 361 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:27.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/3022907929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 msgr2=0x7f1c780f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/3022907929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c780f69b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1c6c004760 tx=0x7f1c6c009ad0).stop 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/3022907929 shutdown_connections 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/3022907929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c780f69b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/3022907929 >> 172.21.15.192:0/3022907929 conn(0x7f1c780f1cd0 msgr2=0x7f1c780f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/3022907929 shutdown_connections 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/3022907929 wait complete. 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 Processor -- start 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.576+0000 7f1c7f0ad700 1 -- start start 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7f0ad700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7f0ad700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78069f00 con 0x7f1c780f6590 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7ce49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7ce49700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43246/0 (socket says 172.21.15.192:43246) 2024-02-20T13:52:27.562 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7ce49700 1 -- 172.21.15.192:0/2843023097 learned_addr learned my addr 172.21.15.192:0/2843023097 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:27.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7ce49700 1 -- 172.21.15.192:0/2843023097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c009d70 con 0x7f1c780f6590 2024-02-20T13:52:27.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.577+0000 7f1c7ce49700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1c6c004170 tx=0x7f1c6c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:27.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.578+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c6c00e040 con 0x7f1c780f6590 2024-02-20T13:52:27.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.578+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7806a0a0 con 0x7f1c780f6590 2024-02-20T13:52:27.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.578+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1c6c008ef0 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.578+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c6c014c90 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.578+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7806a530 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.579+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1c6c014420 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.579+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c780591e0 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.579+0000 7f1c757fa700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 0x7f1c68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.579+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1c6c0422d0 con 0x7f1c780f6590 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.579+0000 7f1c77fff700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 0x7f1c68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:27.565 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.580+0000 7f1c77fff700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 0x7f1c68033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1c640096a0 tx=0x7f1c64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:27.568 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.584+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c6c012d00 con 0x7f1c780f6590 2024-02-20T13:52:27.774 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.789+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c780fc080 con 0x7f1c780f6590 2024-02-20T13:52:27.774 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.790+0000 7f1c757fa700 1 -- 172.21.15.192:0/2843023097 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c6c012b20 con 0x7f1c780f6590 2024-02-20T13:52:27.775 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:27.775 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 msgr2=0x7f1c68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 0x7f1c68033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1c640096a0 tx=0x7f1c64006b40).stop 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 msgr2=0x7f1c7806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1c6c004170 tx=0x7f1c6c009ad0).stop 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 shutdown_connections 2024-02-20T13:52:27.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f1c68031090 0x7f1c68033550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:27.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 --2- 172.21.15.192:0/2843023097 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c780f6590 0x7f1c7806ba80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:27.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 >> 172.21.15.192:0/2843023097 conn(0x7f1c780f1cd0 msgr2=0x7f1c780f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:27.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.791+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 shutdown_connections 2024-02-20T13:52:27.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:27.792+0000 7f1c7f0ad700 1 -- 172.21.15.192:0/2843023097 wait complete. 2024-02-20T13:52:27.777 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:28.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:27.095424+0000 mgr.smithi158.gmoobf (mgr.14156) 362 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:28.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:28 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:27.790385+0000 mon.smithi158 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.192:0/2843023097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:29.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:29.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:30.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:30 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:29.095815+0000 mgr.smithi158.gmoobf (mgr.14156) 363 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:31.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- 172.21.15.192:0/4266373019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 msgr2=0x7fda6c0f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 --2- 172.21.15.192:0/4266373019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0f6ec0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fda68009fb0 tx=0x7fda68009b30).stop 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- 172.21.15.192:0/4266373019 shutdown_connections 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 --2- 172.21.15.192:0/4266373019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0f6ec0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- 172.21.15.192:0/4266373019 >> 172.21.15.192:0/4266373019 conn(0x7fda6c0f0390 msgr2=0x7fda6c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- 172.21.15.192:0/4266373019 shutdown_connections 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- 172.21.15.192:0/4266373019 wait complete. 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 Processor -- start 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- start start 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.572+0000 7fda744d2700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0fb910 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda7226e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda7226e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:43266/0 (socket says 172.21.15.192:43266) 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda7226e700 1 -- 172.21.15.192:0/3952031874 learned_addr learned my addr 172.21.15.192:0/3952031874 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:31.558 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda7226e700 1 -- 172.21.15.192:0/3952031874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68005040 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.559 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda7226e700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fda68000c00 tx=0x7fda6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda680095f0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fda68007e50 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c0fbab0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda68014e70 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.573+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c0fbf40 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.574+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fda680146b0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.560 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.574+0000 7fda62ffd700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 0x7fda580335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:31.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.574+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fda68041be0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.574+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c0591e0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.575+0000 7fda71a6d700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 0x7fda580335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:31.561 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.575+0000 7fda71a6d700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 0x7fda580335a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:31.564 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.579+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fda6801b2c0 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.787+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda6c0fc640 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.772 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.788+0000 7fda62ffd700 1 -- 172.21.15.192:0/3952031874 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda6802c090 con 0x7fda6c0f6aa0 2024-02-20T13:52:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:31.773 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:31.774 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 msgr2=0x7fda580335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:31.774 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 0x7fda580335a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c006b40).stop 2024-02-20T13:52:31.774 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 msgr2=0x7fda6c0fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fda68000c00 tx=0x7fda6800b040).stop 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 shutdown_connections 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7fda580310e0 0x7fda580335a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 --2- 172.21.15.192:0/3952031874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda6c0f6aa0 0x7fda6c0fd490 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 >> 172.21.15.192:0/3952031874 conn(0x7fda6c0f0390 msgr2=0x7fda6c0f4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 shutdown_connections 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:31.790+0000 7fda744d2700 1 -- 172.21.15.192:0/3952031874 wait complete. 2024-02-20T13:52:31.775 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:32.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:32 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:31.096219+0000 mgr.smithi158.gmoobf (mgr.14156) 364 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:32.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:32 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:31.788408+0000 mon.smithi158 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.192:0/3952031874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:33.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:33.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:34.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:34 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:33.096599+0000 mgr.smithi158.gmoobf (mgr.14156) 365 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:35.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.328+0000 7f52afe0e700 1 -- 172.21.15.192:0/2507702791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 msgr2=0x7f52a80f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.328+0000 7f52afe0e700 1 --2- 172.21.15.192:0/2507702791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a80f5e80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f529c009fb0 tx=0x7f529c009b30).stop 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.328+0000 7f52afe0e700 1 -- 172.21.15.192:0/2507702791 shutdown_connections 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.328+0000 7f52afe0e700 1 --2- 172.21.15.192:0/2507702791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a80f5e80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.328+0000 7f52afe0e700 1 -- 172.21.15.192:0/2507702791 >> 172.21.15.192:0/2507702791 conn(0x7f52a80f0390 msgr2=0x7f52a80f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 -- 172.21.15.192:0/2507702791 shutdown_connections 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 -- 172.21.15.192:0/2507702791 wait complete. 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 Processor -- start 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 -- start start 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.329+0000 7f52afe0e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8105bf0 con 0x7f52a80f5a60 2024-02-20T13:52:35.316 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52adbaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52adbaa700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40208/0 (socket says 172.21.15.192:40208) 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52adbaa700 1 -- 172.21.15.192:0/158273456 learned_addr learned my addr 172.21.15.192:0/158273456 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52adbaa700 1 -- 172.21.15.192:0/158273456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c005040 con 0x7f52a80f5a60 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52adbaa700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f529c0089f0 tx=0x7f529c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529c009320 con 0x7f52a80f5a60 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a8105df0 con 0x7f52a80f5a60 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f529c013ec0 con 0x7f52a80f5a60 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529c014d30 con 0x7f52a80f5a60 2024-02-20T13:52:35.317 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.330+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a8106280 con 0x7f52a80f5a60 2024-02-20T13:52:35.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.331+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f529c0095f0 con 0x7f52a80f5a60 2024-02-20T13:52:35.318 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.331+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a8045bb0 con 0x7f52a80f5a60 2024-02-20T13:52:35.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.331+0000 7f529a7fc700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 0x7f5294033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:35.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.331+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f529c042450 con 0x7f52a80f5a60 2024-02-20T13:52:35.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.332+0000 7f52ad3a9700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 0x7f5294033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:35.319 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.332+0000 7f52ad3a9700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 0x7f5294033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:35.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.336+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f529c01d7f0 con 0x7f52a80f5a60 2024-02-20T13:52:35.527 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.542+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52a80faf80 con 0x7f52a80f5a60 2024-02-20T13:52:35.528 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.543+0000 7f529a7fc700 1 -- 172.21.15.192:0/158273456 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f529c01d7f0 con 0x7f52a80f5a60 2024-02-20T13:52:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:35.528 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:35.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 msgr2=0x7f5294033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:35.529 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 0x7f5294033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).stop 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 msgr2=0x7f52a8105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f529c0089f0 tx=0x7f529c00b040).stop 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 shutdown_connections 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f5294031090 0x7f5294033550 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 --2- 172.21.15.192:0/158273456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52a80f5a60 0x7f52a8105700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 >> 172.21.15.192:0/158273456 conn(0x7f52a80f0390 msgr2=0x7f52a80f0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 shutdown_connections 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:35.545+0000 7f52afe0e700 1 -- 172.21.15.192:0/158273456 wait complete. 2024-02-20T13:52:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:36.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:35.097017+0000 mgr.smithi158.gmoobf (mgr.14156) 366 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:36.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:36 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:35.543580+0000 mon.smithi158 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.192:0/158273456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:37.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:52:37.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:52:38.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:38 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:37.097299+0000 mgr.smithi158.gmoobf (mgr.14156) 367 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 -- 172.21.15.192:0/105871069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 msgr2=0x7f7db00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/105871069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db00f56f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7dac009fb0 tx=0x7f7dac009b30).stop 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 -- 172.21.15.192:0/105871069 shutdown_connections 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/105871069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db00f56f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 -- 172.21.15.192:0/105871069 >> 172.21.15.192:0/105871069 conn(0x7f7db00f0a30 msgr2=0x7f7db00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 -- 172.21.15.192:0/105871069 shutdown_connections 2024-02-20T13:52:39.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.167+0000 7f7db6cc9700 1 -- 172.21.15.192:0/105871069 wait complete. 2024-02-20T13:52:39.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db6cc9700 1 Processor -- start 2024-02-20T13:52:39.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db6cc9700 1 -- start start 2024-02-20T13:52:39.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db6cc9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db6cc9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0060320 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db5cc7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db5cc7700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.192:40220/0 (socket says 172.21.15.192:40220) 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db5cc7700 1 -- 172.21.15.192:0/864211195 learned_addr learned my addr 172.21.15.192:0/864211195 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db5cc7700 1 -- 172.21.15.192:0/864211195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac005040 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.168+0000 7f7db5cc7700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f7dac0089f0 tx=0x7f7dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dac009320 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7dac013ec0 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db0060520 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dac014d30 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db0109a20 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.169+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f7dac0095f0 con 0x7f7db00f52d0 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.170+0000 7f7da67fc700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 0x7f7d9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:52:39.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.170+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db0045bb0 con 0x7f7db00f52d0 2024-02-20T13:52:39.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.170+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7dac042390 con 0x7f7db00f52d0 2024-02-20T13:52:39.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.170+0000 7f7db54c6700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 0x7f7d9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:52:39.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.170+0000 7f7db54c6700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 0x7f7d9c033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7da00096a0 tx=0x7f7da0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:52:39.160 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.175+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7dac011290 con 0x7f7db00f52d0 2024-02-20T13:52:39.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.383+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7db010a240 con 0x7f7db00f52d0 2024-02-20T13:52:39.369 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.384+0000 7f7da67fc700 1 -- 172.21.15.192:0/864211195 <== mon.0 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7dac011290 con 0x7f7db00f52d0 2024-02-20T13:52:39.369 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-20T13:52:39.370 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"67c4c1b6-cff5-11ee-95bb-87774f69a715","modified":"2024-02-20T13:39:58.135763Z","created":"2024-02-20T13:39:58.135763Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi158","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:52:39.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 msgr2=0x7f7d9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:39.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 0x7f7d9c033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7da00096a0 tx=0x7f7da0006b40).stop 2024-02-20T13:52:39.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 msgr2=0x7f7db0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:52:39.371 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f7dac0089f0 tx=0x7f7dac00b040).stop 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 shutdown_connections 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:6800/770403169,v1:172.21.15.158:6801/770403169] conn(0x7f7d9c031090 0x7f7d9c033550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 --2- 172.21.15.192:0/864211195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7db00f52d0 0x7f7db0109370 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 >> 172.21.15.192:0/864211195 conn(0x7f7db00f0a30 msgr2=0x7f7db00f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.385+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 shutdown_connections 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:2024-02-20T13:52:39.386+0000 7f7db6cc9700 1 -- 172.21.15.192:0/864211195 wait complete. 2024-02-20T13:52:39.372 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-20T13:52:39.985 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-20T13:52:39.986 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T13:52:39.986 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:52:40.011 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:52:40.029 INFO:tasks.cephadm:Stopping all daemons... 2024-02-20T13:52:40.029 INFO:tasks.cephadm.mon.smithi158:Stopping mon.smithi158... 2024-02-20T13:52:40.029 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158 2024-02-20T13:52:40.531 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 systemd[1]: Stopping Ceph mon.smithi158 for 67c4c1b6-cff5-11ee-95bb-87774f69a715... 2024-02-20T13:52:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 podman[44750]: Error: no container with name or ID "ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon.smithi158" found: no such container 2024-02-20T13:52:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: cluster 2024-02-20T13:52:39.097588+0000 mgr.smithi158.gmoobf (mgr.14156) 368 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:52:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: audit 2024-02-20T13:52:39.385374+0000 mon.smithi158 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.192:0/864211195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:52:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:52:40.345+0000 7f3705fd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi158 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T13:52:40.532 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158[38310]: debug 2024-02-20T13:52:40.345+0000 7f3705fd3700 -1 mon.smithi158@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T13:52:41.031 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:Feb 20 13:52:40 smithi158 bash[44792]: ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715-mon-smithi158 2024-02-20T13:52:41.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi158.service' 2024-02-20T13:52:41.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T13:52:41.173 INFO:tasks.cephadm.mon.smithi158:Stopped mon.smithi158 2024-02-20T13:52:41.173 INFO:tasks.cephadm.mon.smithi192:Stopping mon.smithi192... 2024-02-20T13:52:41.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192 2024-02-20T13:52:41.212 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192.service: Unit ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192.service not loaded. 2024-02-20T13:52:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-20T13:52:41.214 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi192" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 5: 'sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192' 2024-02-20T13:52:41.215 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T13:52:41.217 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi158/crash 2024-02-20T13:52:41.218 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/crash -- . 2024-02-20T13:52:41.259 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi192/crash 2024-02-20T13:52:41.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/crash -- . 2024-02-20T13:52:41.297 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-20T13:52:41.297 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-02-20T13:52:41.298 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T13:52:41.298 DEBUG:teuthology.orchestra.run.smithi158:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T13:52:41.330 INFO:teuthology.orchestra.run.smithi158.stderr:grep: /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph.log: No such file or directory 2024-02-20T13:52:41.331 INFO:tasks.cephadm:Compressing logs... 2024-02-20T13:52:41.332 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T13:52:41.374 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T13:52:41.399 INFO:teuthology.orchestra.run.smithi158.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-20T13:52:41.399 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T13:52:41.399 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T13:52:41.400 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mon.smithi158.log 2024-02-20T13:52:41.400 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mgr.smithi158.gmoobf.log 2024-02-20T13:52:41.400 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mon.smithi158.log: /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mgr.smithi158.gmoobf.log: 90.1% -- replaced with /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T13:52:41.405 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-20T13:52:41.406 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T13:52:41.406 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T13:52:41.407 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/cephadm.log: 88.4% -- replaced with /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T13:52:41.408 INFO:teuthology.orchestra.run.smithi158.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T13:52:41.411 INFO:teuthology.orchestra.run.smithi192.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T13:52:41.412 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-02-20T13:52:41.412 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m0.018s 2024-02-20T13:52:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m0.016s 2024-02-20T13:52:41.413 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.012s 2024-02-20T13:52:41.493 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mgr.smithi158.gmoobf.log: file size changed while zipping 2024-02-20T13:52:41.493 INFO:teuthology.orchestra.run.smithi158.stderr: 92.8% -- replaced with /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mgr.smithi158.gmoobf.log.gz 2024-02-20T13:52:41.565 INFO:teuthology.orchestra.run.smithi158.stderr: 90.6% -- replaced with /var/log/ceph/67c4c1b6-cff5-11ee-95bb-87774f69a715/ceph-mon.smithi158.log.gz 2024-02-20T13:52:41.567 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-20T13:52:41.567 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m0.178s 2024-02-20T13:52:41.567 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m0.265s 2024-02-20T13:52:41.568 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m0.025s 2024-02-20T13:52:41.568 INFO:tasks.cephadm:Archiving logs... 2024-02-20T13:52:41.568 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi158/log 2024-02-20T13:52:41.569 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T13:52:41.691 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi192/log 2024-02-20T13:52:41.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T13:52:41.724 INFO:tasks.cephadm:Removing cluster... 2024-02-20T13:52:41.724 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 --force 2024-02-20T13:56:23.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67c4c1b6-cff5-11ee-95bb-87774f69a715 --force 2024-02-20T13:56:23.913 INFO:tasks.cephadm:Removing cephadm ... 2024-02-20T13:56:23.913 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T13:56:23.933 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T13:56:23.952 INFO:tasks.cephadm:Teardown complete 2024-02-20T13:56:23.952 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 5: 'sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192' 2024-02-20T13:56:24.182 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fcf3ee8eb4014a7a8d2ff8c03bca3e8a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 5: 'sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192' 2024-02-20T13:56:24.187 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T13:56:24.199 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T13:56:24.214 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T13:56:24.214 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T13:56:24.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T13:56:24.308 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T13:56:24.317 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T13:56:24.318 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:56:24.320 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:56:24.336 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2024-02-20T13:56:24.337 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -12us[ -12us] +/- 104ms 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -698us[ -694us] +/- 51ms 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +938us[ +938us] +/- 72ms 2024-02-20T13:56:24.341 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:56:24.343 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:56:24.343 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-02-20T13:56:24.343 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 +9965ns[+9965ns] +/- 104ms 2024-02-20T13:56:24.343 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -670us[ -675us] +/- 51ms 2024-02-20T13:56:24.344 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +943us[ +939us] +/- 72ms 2024-02-20T13:56:24.344 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:56:24.344 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T13:56:24.353 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T13:56:24.354 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T13:56:24.369 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T13:56:24.445 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T13:56:24.509 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:56:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:56:24.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:56:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:56:24.568 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T13:56:24.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T13:56:24.590 INFO:teuthology.task.internal:Duration was 1649.755905 seconds 2024-02-20T13:56:24.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T13:56:24.621 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T13:56:24.621 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T13:56:24.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T13:56:24.667 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:56:24.669 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:56:25.041 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-20T13:56:25.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2024-02-20T13:56:25.042 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T13:56:25.065 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-02-20T13:56:25.065 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T13:56:25.087 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-20T13:56:25.087 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T13:56:25.107 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T13:56:25.176 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-20T13:56:25.177 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T13:56:25.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T13:56:25.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T13:56:25.281 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T13:56:25.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T13:56:25.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T13:56:25.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T13:56:25.352 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:56:25.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:56:25.379 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2024-02-20T13:56:25.402 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-02-20T13:56:25.422 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:56:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:56:25.473 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:56:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:56:25.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T13:56:25.500 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T13:56:25.502 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi158 2024-02-20T13:56:25.502 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T13:56:25.555 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738/remote/smithi192 2024-02-20T13:56:25.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T13:56:25.594 INFO:teuthology.task.internal:Removing archive directory... 2024-02-20T13:56:25.594 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T13:56:25.597 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T13:56:25.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T13:56:25.656 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T13:56:25.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T13:56:25.672 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T13:56:25.672 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T13:56:25.674 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T13:56:25.689 INFO:teuthology.orchestra.run.smithi158.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 13:56 /home/ubuntu/cephtest 2024-02-20T13:56:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 13:56 /home/ubuntu/cephtest 2024-02-20T13:56:25.704 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T13:56:25.714 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T13:56:25.745 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T13:56:25.770 DEBUG:teuthology.nuke:shortname: smithi158 2024-02-20T13:56:25.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:56:25.797 DEBUG:teuthology.nuke:shortname: smithi192 2024-02-20T13:56:25.797 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:56:25.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 13:19:59.454176', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMARbUYPR2E4vAmpXPizT7pEBPn1jeDgO0LZrIFNmAwjmONrV9SoIF1GkMAW8UbBpBfRbZpBlJgYod/GusHIxhk='} 2024-02-20T13:56:25.820 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567738', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 13:19:59.452132', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCLuTYqzb4pdMWLS7oIsjFu51Zgv8dbXQ91B2L30gC5RWw4KLmGKWcfqRDnaRI7ubMCurPcmqBnaZTQHzbVtJ8='} 2024-02-20T13:56:25.851 INFO:teuthology.orchestra.console.smithi158:Power off 2024-02-20T13:56:25.851 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T13:56:25.880 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-02-20T13:56:25.880 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:25.896 INFO:teuthology.orchestra.console.smithi192:Power off 2024-02-20T13:56:25.897 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T13:56:25.907 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-20T13:56:25.921 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-02-20T13:56:25.921 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:25.946 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-02-20T13:56:29.908 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:29.948 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:30.020 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-20T13:56:30.058 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-02-20T13:56:34.024 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:34.060 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:56:34.134 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-02-20T13:56:34.170 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-02-20T13:56:34.236 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-02-20T13:56:34.272 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-02-20T13:56:34.377 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1649.7559053897858 failure_reason: 'Command failed on smithi192 with status 5: ''sudo systemctl stop ceph-67c4c1b6-cff5-11ee-95bb-87774f69a715@mon.smithi192''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fcf3ee8eb4014a7a8d2ff8c03bca3e8a status: fail success: false 2024-02-20T13:56:34.377 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T13:56:34.457 INFO:teuthology.run:FAIL