2023-11-16T13:44:26.376 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T13:44:26.376 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812 --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi --owner scheduled_gabrioux@teuthology -- /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/orig.config.yaml 2023-11-16T13:44:26.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T13:44:26.565 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812 branch: wip-63555-quincy description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: gabrioux@ibm.com first_in_suite: false job_id: '7460812' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-63555-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-63555-quincy sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3535 sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-63555-quincy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 targets: smithi061.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJvIhhtaHs2kljBp2KydiS8aDaamE2L/NVmrcaJ0oiamX5Jo7A9fHWw4xvlHfoe7F2av3aUiEFGmN2C1i7wdZU= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbouQQgrOaglJ3snHsZZY6SjfVoJ5UJPVnHLe0Hg+rXqG/G/VNiDd/XEiDUw9grCYbfoaz8E9i+KhWtnkjqlwI= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_12:07:54 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-16T13:44:26.565 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa; will attempt to use it 2023-11-16T13:44:26.566 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks 2023-11-16T13:44:26.567 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T13:44:26.569 INFO:teuthology.task.internal:Checking packages... 2023-11-16T13:44:26.602 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4' 2023-11-16T13:44:26.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T13:44:26.602 INFO:teuthology.packaging:ref: None 2023-11-16T13:44:26.602 INFO:teuthology.packaging:tag: None 2023-11-16T13:44:26.602 INFO:teuthology.packaging:branch: wip-63555-quincy 2023-11-16T13:44:26.602 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T13:44:26.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-63555-quincy 2023-11-16T13:44:26.847 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-68.gae6ce92c 2023-11-16T13:44:26.847 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T13:44:26.854 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T13:44:26.854 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T13:44:26.860 INFO:teuthology.task.internal:Saving configuration 2023-11-16T13:44:26.884 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T13:44:26.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T13:44:26.918 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 13:39:23.612701', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJvIhhtaHs2kljBp2KydiS8aDaamE2L/NVmrcaJ0oiamX5Jo7A9fHWw4xvlHfoe7F2av3aUiEFGmN2C1i7wdZU='} 2023-11-16T13:44:26.940 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 13:39:23.614296', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbouQQgrOaglJ3snHsZZY6SjfVoJ5UJPVnHLe0Hg+rXqG/G/VNiDd/XEiDUw9grCYbfoaz8E9i+KhWtnkjqlwI='} 2023-11-16T13:44:26.940 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T13:44:26.946 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-16T13:44:26.947 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-16T13:44:26.947 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T13:44:27.049 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f116b02bf70>, signals=[15]) 2023-11-16T13:44:27.049 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T13:44:27.057 INFO:teuthology.task.internal:Opening connections... 2023-11-16T13:44:27.058 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2023-11-16T13:44:27.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:44:27.130 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2023-11-16T13:44:27.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:44:27.200 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T13:44:27.207 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2023-11-16T13:44:27.231 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2023-11-16T13:44:27.232 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-11-16T13:44:27.294 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Stream" 2023-11-16T13:44:27.294 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8" 2023-11-16T13:44:27.294 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-11-16T13:44:27.295 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-16T13:44:27.296 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2023-11-16T13:44:27.296 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T13:44:27.296 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T13:44:27.296 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-16T13:44:27.297 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-11-16T13:44:27.318 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2023-11-16T13:44:27.341 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-11-16T13:44:27.341 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2023-11-16T13:44:27.401 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2023-11-16T13:44:27.401 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8" 2023-11-16T13:44:27.401 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T13:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T13:44:27.403 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-16T13:44:27.403 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-11-16T13:44:27.426 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T13:44:27.436 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T13:44:27.441 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T13:44:27.442 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2023-11-16T13:44:27.445 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2023-11-16T13:44:27.466 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T13:44:27.473 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T13:44:27.473 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2023-11-16T13:44:27.507 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2023-11-16T13:44:27.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T13:44:27.593 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T13:44:27.608 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-16T13:44:27.608 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-16T13:44:27.608 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T13:44:27.609 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T13:44:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T13:44:27.629 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-11-16T13:44:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T13:44:27.631 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-11-16T13:44:27.690 INFO:teuthology.orchestra.run.smithi061.stdout:4.18.0-489.el8.x86_64 2023-11-16T13:44:27.691 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.18.0-489.el8.x86_64 2023-11-16T13:44:27.691 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum install -y kernel 2023-11-16T13:44:27.693 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-489.el8.x86_64 2023-11-16T13:44:27.693 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-489.el8.x86_64 2023-11-16T13:44:27.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2023-11-16T13:44:49.739 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 1.7 MB/s | 34 MB 00:19 2023-11-16T13:44:58.222 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-16T13:45:06.181 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 58 kB/s | 18 kB 00:00 2023-11-16T13:45:08.164 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 3.8 kB/s | 6.9 kB 00:01 2023-11-16T13:45:08.449 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-16T13:45:09.474 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - AppStream 884 kB/s | 34 MB 00:39 2023-11-16T13:45:10.108 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-16T13:45:13.900 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-11-16T13:45:17.524 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-16T13:45:17.524 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-16T13:45:17.801 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:45:17.801 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:45:17.802 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:45:17.803 INFO:teuthology.orchestra.run.smithi112.stdout:Install 3 Packages 2023-11-16T13:45:17.803 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:17.803 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 89 M 2023-11-16T13:45:17.803 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 96 M 2023-11-16T13:45:17.804 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:45:18.132 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 53 MB 00:02 2023-11-16T13:45:19.026 INFO:teuthology.orchestra.run.smithi112.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 9.8 MB/s | 10 MB 00:01 2023-11-16T13:45:20.068 INFO:teuthology.orchestra.run.smithi112.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2023-11-16T13:45:20.918 INFO:teuthology.orchestra.run.smithi112.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2023-11-16T13:45:20.919 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:45:20.919 INFO:teuthology.orchestra.run.smithi112.stdout:Total 29 MB/s | 89 MB 00:03 2023-11-16T13:45:21.589 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:45:21.994 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:45:29.420 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:45:29.421 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2023-11-16T13:45:29.421 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:45:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:45:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:45:30.466 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:45:32.944 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 603 kB/s | 24 kB 00:00 2023-11-16T13:45:33.893 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:45:34.163 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:36.701 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-16T13:45:36.702 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-16T13:45:37.006 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:45:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:Install 3 Packages 2023-11-16T13:45:37.008 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:45:37.008 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 89 M 2023-11-16T13:45:37.008 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 96 M 2023-11-16T13:45:37.008 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:45:38.120 INFO:teuthology.orchestra.run.smithi061.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 9.4 MB/s | 10 MB 00:01 2023-11-16T13:45:39.212 INFO:teuthology.orchestra.run.smithi061.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2023-11-16T13:45:39.829 INFO:teuthology.orchestra.run.smithi061.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-11-16T13:45:39.829 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:45:39.829 INFO:teuthology.orchestra.run.smithi061.stdout:Total 32 MB/s | 89 MB 00:02 2023-11-16T13:45:40.544 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:45:40.990 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:45:40.990 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:45:42.278 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:45:42.298 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:45:45.214 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:45:45.810 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:45:49.518 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:45:49.901 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:45:53.573 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:45:53.844 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:45:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:45:57.117 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:45:57.118 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:57.118 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:45:57.118 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-16T13:45:57.118 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-16T13:45:57.119 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:57.119 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:45:57.256 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T13:45:57.476 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T13:45:57.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T13:45:57.477 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-16T13:45:57.477 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T13:45:57.477 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T13:45:57.478 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-16T13:45:57.478 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T13:45:57.478 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T13:45:57.686 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T13:45:57.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T13:45:57.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2023-11-16T13:45:57.719 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2023-11-16T13:45:57.719 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.02 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 129.el8 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Group : System Environment/Base 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 9265586 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-16T13:45:57.720 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Relocations : (not relocatable) 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : CentOS BuildSystem 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2023-11-16T13:45:57.721 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:45:57.722 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T13:45:57.722 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T13:45:57.723 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 4.18.0-521.el8.x86_64 2023-11-16T13:45:57.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T13:45:58.974 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2023-11-16T13:46:00.754 INFO:teuthology.orchestra.run.smithi112.stderr:done 2023-11-16T13:46:00.755 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-11-16T13:46:00.783 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.orSCGLftXz 2023-11-16T13:46:00.784 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.orSCGLftXz 2023-11-16T13:46:00.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.orSCGLftXz 2023-11-16T13:46:01.012 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.orSCGLftXz is 6KB 2023-11-16T13:46:01.060 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.orSCGLftXz 2023-11-16T13:46:01.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T13:46:01.159 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-16T13:46:01.159 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-16T13:46:01.159 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-16T13:46:01.159 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-16T13:46:01.160 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-16T13:46:01.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2023-11-16T13:46:17.117 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:46:17.141 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:46:17.665 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:46:17.665 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-16T13:46:17.666 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:46:18.529 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:46:18.680 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T13:46:18.910 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T13:46:18.910 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-16T13:46:18.910 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T13:46:18.911 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T13:46:18.911 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-16T13:46:18.911 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T13:46:18.911 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T13:46:19.129 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T13:46:19.129 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T13:46:19.130 DEBUG:teuthology.orchestra.run.smithi061:> sudo rpm -qi grub2-tools 2023-11-16T13:46:19.167 INFO:teuthology.orchestra.run.smithi061.stdout:Name : grub2-tools 2023-11-16T13:46:19.167 INFO:teuthology.orchestra.run.smithi061.stdout:Epoch : 1 2023-11-16T13:46:19.167 INFO:teuthology.orchestra.run.smithi061.stdout:Version : 2.02 2023-11-16T13:46:19.167 INFO:teuthology.orchestra.run.smithi061.stdout:Release : 129.el8 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: x86_64 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Group : System Environment/Base 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Size : 9265586 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:License : GPLv3+ 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Relocations : (not relocatable) 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Packager : CentOS BuildSystem 2023-11-16T13:46:19.168 INFO:teuthology.orchestra.run.smithi061.stdout:Vendor : CentOS 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:Summary : Support tools for GRUB. 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:Description : 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:hardware devices. 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:46:19.169 INFO:teuthology.orchestra.run.smithi061.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T13:46:19.170 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T13:46:19.170 INFO:teuthology.task.kernel:Updating grub on smithi061 to boot 4.18.0-521.el8.x86_64 2023-11-16T13:46:19.170 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T13:46:20.736 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2023-11-16T13:46:22.605 INFO:teuthology.orchestra.run.smithi061.stderr:done 2023-11-16T13:46:22.609 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2023-11-16T13:46:22.631 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.kvLd7Pe4Zr 2023-11-16T13:46:22.631 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kvLd7Pe4Zr 2023-11-16T13:46:22.714 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.kvLd7Pe4Zr 2023-11-16T13:46:22.853 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.kvLd7Pe4Zr is 6KB 2023-11-16T13:46:22.901 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.kvLd7Pe4Zr 2023-11-16T13:46:22.919 DEBUG:teuthology.orchestra.run.smithi061:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T13:46:22.985 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-16T13:46:22.986 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-16T13:46:22.986 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-16T13:46:22.986 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-16T13:46:22.987 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-16T13:46:23.133 DEBUG:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2023-11-16T13:46:31.323 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-16T13:46:31.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:46:31.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:46:53.156 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-16T13:46:53.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:46:53.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:47:05.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-11-16T13:47:14.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:47:14.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:47:27.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-11-16T13:47:33.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-11-16T13:47:36.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:47:36.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:47:47.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:47:47.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:47:48.582 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-11-16T13:47:49.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:47:49.273 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-16T13:47:49.274 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-11-16T13:47:49.291 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-521.el8.x86_64 2023-11-16T13:47:49.291 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-16T13:47:49.291 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T13:47:49.292 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-16T13:47:50.293 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-16T13:47:50.293 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T13:47:50.366 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2023-11-16T13:47:50.378 DEBUG:teuthology.parallel:result is None 2023-11-16T13:47:55.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-11-16T13:48:07.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:48:07.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:48:07.482 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-11-16T13:48:07.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:48:07.759 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-16T13:48:07.759 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-11-16T13:48:07.778 INFO:teuthology.orchestra.run.smithi061.stdout:4.18.0-521.el8.x86_64 2023-11-16T13:48:07.778 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-16T13:48:07.778 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T13:48:07.778 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-16T13:48:08.779 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-16T13:48:08.780 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T13:48:09.094 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2023-11-16T13:48:09.123 DEBUG:teuthology.parallel:result is None 2023-11-16T13:48:09.123 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T13:48:09.134 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T13:48:09.134 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T13:48:09.176 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T13:48:09.539 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T13:48:09.545 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T13:48:09.554 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T13:48:09.554 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T13:48:09.590 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T13:48:10.016 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T13:48:10.026 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T13:48:10.027 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T13:48:10.052 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T13:48:10.094 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T13:48:10.105 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T13:48:10.107 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T13:48:10.118 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T13:48:10.119 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T13:48:10.127 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T13:48:10.128 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T13:48:10.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T13:48:10.178 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T13:48:10.188 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T13:48:10.188 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T13:48:10.200 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T13:48:10.237 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T13:48:10.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T13:48:10.410 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T13:48:10.456 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T13:48:10.498 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T13:48:10.498 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T13:48:10.565 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T13:48:10.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T13:48:10.671 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T13:48:10.758 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T13:48:10.803 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T13:48:10.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T13:48:10.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2023-11-16T13:48:10.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2023-11-16T13:48:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T13:48:10.983 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T13:48:11.419 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T13:48:11.428 INFO:teuthology.task.internal:Starting timer... 2023-11-16T13:48:11.428 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T13:48:11.449 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T13:48:11.459 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T13:48:11.460 DEBUG:teuthology.orchestra.run.smithi061:> sudo service auditd rotate 2023-11-16T13:48:11.570 INFO:teuthology.orchestra.run.smithi061.stdout:Rotating logs: 2023-11-16T13:48:11.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2023-11-16T13:48:11.668 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2023-11-16T13:48:11.672 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T13:48:11.673 DEBUG:teuthology.orchestra.run.smithi061:> /usr/sbin/getenforce 2023-11-16T13:48:11.705 INFO:teuthology.orchestra.run.smithi061.stdout:Permissive 2023-11-16T13:48:11.706 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2023-11-16T13:48:11.740 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2023-11-16T13:48:11.741 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi061.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2023-11-16T13:48:11.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T13:48:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T13:48:11.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T13:48:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T13:48:11.808 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T13:48:11.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo /usr/sbin/setenforce permissive 2023-11-16T13:48:11.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2023-11-16T13:48:11.922 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T13:48:11.934 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-16T13:48:11.935 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T13:48:11.950 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T13:48:11.951 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T13:55:54.951 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2023-11-16T13:55:54.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:55:54.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:55:55.025 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-11-16T13:55:55.095 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-11-16T13:55:55.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:55:55.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T13:55:55.166 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-11-16T13:55:55.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-11-16T13:55:55.238 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T13:55:55.252 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T13:55:55.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T13:55:55.253 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T13:55:55.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T13:55:55.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T13:55:55.288 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T13:55:55.303 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T13:55:55.310 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T13:55:55.324 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T13:55:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: ntpd: command not found 2023-11-16T13:55:55.374 INFO:teuthology.orchestra.run.smithi061.stdout:506 Cannot talk to daemon 2023-11-16T13:55:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T13:55:55.389 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2023-11-16T13:55:55.400 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2023-11-16T13:55:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T13:55:55.414 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T13:55:55.427 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T13:55:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-11-16T13:55:55.485 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T13:55:55.485 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-11-16T13:55:55.486 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.486 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.486 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.486 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.491 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-11-16T13:55:55.494 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T13:55:55.494 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-11-16T13:55:55.495 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.495 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.495 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.495 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T13:55:55.496 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T13:55:55.507 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T13:55:55.508 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi061.front.sepia.ceph.com 2023-11-16T13:55:55.508 DEBUG:teuthology.orchestra.run.smithi061:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T13:55:55.509 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi112.front.sepia.ceph.com 2023-11-16T13:55:55.509 DEBUG:teuthology.orchestra.run.smithi112:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T13:55:56.416 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:47 ago on Thu 16 Nov 2023 01:54:09 PM UTC. 2023-11-16T13:55:56.440 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:37 ago on Thu 16 Nov 2023 01:54:19 PM UTC. 2023-11-16T13:55:58.501 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:55:58.501 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout:Resetting modules: 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:55:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:55:58.722 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout:Resetting modules: 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:55:58.724 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:55:58.734 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:55:58.974 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:55:59.659 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:50 ago on Thu 16 Nov 2023 01:54:09 PM UTC. 2023-11-16T13:55:59.957 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:40 ago on Thu 16 Nov 2023 01:54:19 PM UTC. 2023-11-16T13:56:01.749 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:56:01.926 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:56:01.926 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:01.926 INFO:teuthology.orchestra.run.smithi112.stdout:Installing group/module packages: 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-16T13:56:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:Installing module profiles: 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools/common 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:Enabling module streams: 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools rhel8 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2023-11-16T13:56:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 27 M 2023-11-16T13:56:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 89 M 2023-11-16T13:56:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:56:02.217 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:56:02.409 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:56:02.409 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:02.409 INFO:teuthology.orchestra.run.smithi061.stdout:Installing group/module packages: 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-16T13:56:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:Installing weak dependencies: 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:Installing module profiles: 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools/common 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling module streams: 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools rhel8 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout:Install 13 Packages 2023-11-16T13:56:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:02.412 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 27 M 2023-11-16T13:56:02.412 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 89 M 2023-11-16T13:56:02.412 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:56:03.170 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.1 MB/s | 750 kB 00:00 2023-11-16T13:56:03.237 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.4 MB/s | 1.0 MB 00:00 2023-11-16T13:56:03.271 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2023-11-16T13:56:03.371 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.8 MB/s | 241 kB 00:00 2023-11-16T13:56:03.488 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.5 MB/s | 178 kB 00:00 2023-11-16T13:56:03.588 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2023-11-16T13:56:03.672 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.8 MB/s | 3.9 MB 00:00 2023-11-16T13:56:03.697 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 864 kB/s | 94 kB 00:00 2023-11-16T13:56:03.856 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.2 MB/s | 9.6 MB 00:01 2023-11-16T13:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 528 kB/s | 53 kB 00:00 2023-11-16T13:56:04.041 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 7.4 MB/s | 2.5 MB 00:00 2023-11-16T13:56:04.057 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 409 kB/s | 41 kB 00:00 2023-11-16T13:56:04.265 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 14 MB/s | 8.1 MB 00:00 2023-11-16T13:56:04.266 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:56:04.266 INFO:teuthology.orchestra.run.smithi112.stdout:Total 11 MB/s | 27 MB 00:02 2023-11-16T13:56:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.1 MB/s | 750 kB 00:00 2023-11-16T13:56:04.354 INFO:teuthology.orchestra.run.smithi061.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.4 MB/s | 1.0 MB 00:00 2023-11-16T13:56:04.379 INFO:teuthology.orchestra.run.smithi061.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 413 kB/s | 38 kB 00:00 2023-11-16T13:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:56:04.562 INFO:teuthology.orchestra.run.smithi061.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.1 MB/s | 241 kB 00:00 2023-11-16T13:56:04.663 INFO:teuthology.orchestra.run.smithi061.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2023-11-16T13:56:04.763 INFO:teuthology.orchestra.run.smithi061.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2023-11-16T13:56:04.805 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:56:04.808 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:56:04.838 INFO:teuthology.orchestra.run.smithi061.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 8.5 MB/s | 3.9 MB 00:00 2023-11-16T13:56:04.964 INFO:teuthology.orchestra.run.smithi061.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.1 MB/s | 9.6 MB 00:01 2023-11-16T13:56:04.989 INFO:teuthology.orchestra.run.smithi061.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 416 kB/s | 94 kB 00:00 2023-11-16T13:56:05.248 INFO:teuthology.orchestra.run.smithi061.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 8.9 MB/s | 2.5 MB 00:00 2023-11-16T13:56:05.273 INFO:teuthology.orchestra.run.smithi061.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 185 kB/s | 53 kB 00:00 2023-11-16T13:56:05.366 INFO:teuthology.orchestra.run.smithi061.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 15 MB/s | 8.1 MB 00:00 2023-11-16T13:56:05.390 INFO:teuthology.orchestra.run.smithi061.stdout:(13/13): yajl-2.1.0-12.el8.x86_64.rpm 289 kB/s | 41 kB 00:00 2023-11-16T13:56:05.391 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:56:05.391 INFO:teuthology.orchestra.run.smithi061.stdout:Total 8.9 MB/s | 27 MB 00:02 2023-11-16T13:56:05.502 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:56:05.670 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:56:05.693 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:56:05.693 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-16T13:56:05.897 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-16T13:56:05.987 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:56:05.991 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:56:06.098 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-16T13:56:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T13:56:06.689 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:56:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-16T13:56:06.938 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-16T13:56:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-16T13:56:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-16T13:56:07.556 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T13:56:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T13:56:08.135 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-16T13:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-16T13:56:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-16T13:56:08.619 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-16T13:56:09.013 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-16T13:56:09.154 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T13:56:09.380 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-16T13:56:09.595 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-16T13:56:09.879 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-16T13:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T13:56:10.085 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-16T13:56:10.271 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T13:56:10.302 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-16T13:56:10.691 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T13:56:10.692 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-16T13:56:10.692 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-16T13:56:10.693 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-16T13:56:10.693 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T13:56:10.694 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-16T13:56:10.694 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T13:56:10.694 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-16T13:56:10.694 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-16T13:56:10.695 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-16T13:56:10.695 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-16T13:56:10.695 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T13:56:10.695 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-16T13:56:11.235 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T13:56:11.457 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-16T13:56:11.513 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-16T13:56:11.513 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:11.513 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:56:11.513 INFO:teuthology.orchestra.run.smithi112.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-16T13:56:11.514 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:11.515 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:56:11.802 DEBUG:teuthology.parallel:result is None 2023-11-16T13:56:12.232 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-16T13:56:12.693 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T13:56:12.694 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-16T13:56:13.538 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-16T13:56:13.538 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-16T13:56:13.539 INFO:teuthology.orchestra.run.smithi061.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-16T13:56:13.540 INFO:teuthology.orchestra.run.smithi061.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-16T13:56:13.540 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:13.540 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:56:13.777 DEBUG:teuthology.parallel:result is None 2023-11-16T13:56:13.777 INFO:teuthology.run_tasks:Running task install... 2023-11-16T13:56:13.786 DEBUG:teuthology.task.install:project ceph 2023-11-16T13:56:13.787 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4'}} 2023-11-16T13:56:13.787 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4'} 2023-11-16T13:56:13.787 INFO:teuthology.task.install:Using flavor: default 2023-11-16T13:56:13.793 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T13:56:13.793 INFO:teuthology.task.install:extra packages: [] 2023-11-16T13:56:13.794 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4', 'tag': None, 'wait_for_package': False} 2023-11-16T13:56:13.794 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T13:56:13.794 INFO:teuthology.packaging:ref: None 2023-11-16T13:56:13.794 INFO:teuthology.packaging:tag: None 2023-11-16T13:56:13.794 INFO:teuthology.packaging:branch: pacific 2023-11-16T13:56:13.795 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T13:56:13.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-11-16T13:56:13.797 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4', 'tag': None, 'wait_for_package': False} 2023-11-16T13:56:13.798 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T13:56:13.798 INFO:teuthology.packaging:ref: None 2023-11-16T13:56:13.798 INFO:teuthology.packaging:tag: None 2023-11-16T13:56:13.798 INFO:teuthology.packaging:branch: pacific 2023-11-16T13:56:13.798 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T13:56:13.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-11-16T13:56:14.058 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/ 2023-11-16T13:56:14.059 INFO:teuthology.task.install.rpm:Package version is 16.2.14-180.g9513c28f 2023-11-16T13:56:14.113 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/ 2023-11-16T13:56:14.113 INFO:teuthology.task.install.rpm:Package version is 16.2.14-180.g9513c28f 2023-11-16T13:56:14.227 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T13:56:14.228 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T13:56:14.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T13:56:14.242 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/9513c28fe2d0c3ab65a3cff29b7944305ca42a95/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T13:56:14.242 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T13:56:14.242 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T13:56:14.257 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-16T13:56:14.258 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T13:56:14.258 INFO:teuthology.packaging:ref: None 2023-11-16T13:56:14.258 INFO:teuthology.packaging:tag: None 2023-11-16T13:56:14.258 INFO:teuthology.packaging:branch: pacific 2023-11-16T13:56:14.258 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T13:56:14.259 DEBUG:teuthology.orchestra.run.smithi061:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T13:56:14.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-16T13:56:14.276 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T13:56:14.276 INFO:teuthology.packaging:ref: None 2023-11-16T13:56:14.277 INFO:teuthology.packaging:tag: None 2023-11-16T13:56:14.277 INFO:teuthology.packaging:branch: pacific 2023-11-16T13:56:14.277 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T13:56:14.278 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T13:56:14.328 DEBUG:teuthology.orchestra.run.smithi061:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T13:56:14.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T13:56:14.409 DEBUG:teuthology.orchestra.run.smithi061:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T13:56:14.430 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T13:56:14.477 INFO:teuthology.orchestra.run.smithi061.stdout:check_obsoletes = 1 2023-11-16T13:56:14.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum clean all 2023-11-16T13:56:14.499 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2023-11-16T13:56:14.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2023-11-16T13:56:14.761 INFO:teuthology.orchestra.run.smithi061.stdout:76 files removed 2023-11-16T13:56:14.767 INFO:teuthology.orchestra.run.smithi112.stdout:76 files removed 2023-11-16T13:56:14.789 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-radosgw 2023-11-16T13:56:14.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2023-11-16T13:56:15.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph packages for x86_64 153 kB/s | 76 kB 00:00 2023-11-16T13:56:15.860 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 115 kB/s | 76 kB 00:00 2023-11-16T13:56:16.135 INFO:teuthology.orchestra.run.smithi061.stdout:ceph noarch packages 41 kB/s | 15 kB 00:00 2023-11-16T13:56:16.240 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2023-11-16T13:56:16.399 INFO:teuthology.orchestra.run.smithi061.stdout:ceph source packages 6.1 kB/s | 1.4 kB 00:00 2023-11-16T13:56:16.557 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 5.0 kB/s | 1.4 kB 00:00 2023-11-16T13:56:18.124 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-16T13:56:18.541 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 17 MB/s | 34 MB 00:01 2023-11-16T13:56:26.575 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 53 MB 00:01 2023-11-16T13:56:26.950 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 24 MB/s | 53 MB 00:02 2023-11-16T13:56:35.009 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-16T13:56:41.376 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2023-11-16T13:56:41.402 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - PowerTools 5.1 MB/s | 6.6 MB 00:01 2023-11-16T13:56:41.402 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-11-16T13:56:41.402 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2023-11-16T13:56:41.402 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2023-11-16T13:56:41.403 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2023-11-16T13:56:41.403 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - PowerTools 5.5 MB/s | 6.6 MB 00:01 2023-11-16T13:56:41.403 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-11-16T13:56:41.403 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2023-11-16T13:56:44.488 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 143 kB/s | 24 kB 00:00 2023-11-16T13:56:44.495 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 524 kB/s | 24 kB 00:00 2023-11-16T13:56:49.061 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:16.2.14-180.g9513c28f.el8 ceph 12 M 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.7 M 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 4.2 M 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:16.2.14-180.g9513c28f.el8 ceph 6.1 M 2023-11-16T13:56:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:16.2.14-180.g9513c28f.el8 ceph 24 M 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:16.2.14-180.g9513c28f.el8 ceph 26 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 824 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 494 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.8 M 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 45 k 2023-11-16T13:56:49.065 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:16.2.14-180.g9513c28f.el8 ceph 99 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:16.2.14-180.g9513c28f.el8 ceph 215 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:16.2.14-180.g9513c28f.el8 ceph 390 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 369 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:16.2.14-180.g9513c28f.el8 ceph 114 k 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout:Install 21 Packages 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2023-11-16T13:56:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:56:49.067 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 57 M 2023-11-16T13:56:49.067 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:56:49.349 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw x86_64 2:16.2.14-180.g9513c28f.el8 ceph 12 M 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrading: 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.7 M 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 4.2 M 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base x86_64 2:16.2.14-180.g9513c28f.el8 ceph 6.1 M 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common x86_64 2:16.2.14-180.g9513c28f.el8 ceph 24 M 2023-11-16T13:56:49.352 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux x86_64 2:16.2.14-180.g9513c28f.el8 ceph 26 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 824 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 494 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.8 M 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 45 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common x86_64 2:16.2.14-180.g9513c28f.el8 ceph 99 k 2023-11-16T13:56:49.353 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs x86_64 2:16.2.14-180.g9513c28f.el8 ceph 215 k 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados x86_64 2:16.2.14-180.g9513c28f.el8 ceph 390 k 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 369 k 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw x86_64 2:16.2.14-180.g9513c28f.el8 ceph 114 k 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout:Install 21 Packages 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrade 2 Packages 2023-11-16T13:56:49.354 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:56:49.355 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 57 M 2023-11-16T13:56:49.355 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:56:50.406 INFO:teuthology.orchestra.run.smithi061.stdout:(1/23): ceph-base-16.2.14-180.g9513c28f.el8.x86 6.5 MB/s | 6.1 MB 00:00 2023-11-16T13:56:50.464 INFO:teuthology.orchestra.run.smithi061.stdout:(2/23): ceph-selinux-16.2.14-180.g9513c28f.el8. 441 kB/s | 26 kB 00:00 2023-11-16T13:56:50.640 INFO:teuthology.orchestra.run.smithi061.stdout:(3/23): libcephfs2-16.2.14-180.g9513c28f.el8.x8 4.6 MB/s | 824 kB 00:00 2023-11-16T13:56:50.781 INFO:teuthology.orchestra.run.smithi061.stdout:(4/23): libradosstriper1-16.2.14-180.g9513c28f. 3.4 MB/s | 494 kB 00:00 2023-11-16T13:56:51.707 INFO:teuthology.orchestra.run.smithi061.stdout:(5/23): librgw2-16.2.14-180.g9513c28f.el8.x86_6 4.1 MB/s | 3.8 MB 00:00 2023-11-16T13:56:51.755 INFO:teuthology.orchestra.run.smithi112.stdout:(1/23): ceph-common-16.2.14-180.g9513c28f.el8.x 10 MB/s | 24 MB 00:02 2023-11-16T13:56:51.757 INFO:teuthology.orchestra.run.smithi061.stdout:(6/23): python3-ceph-argparse-16.2.14-180.g9513 905 kB/s | 45 kB 00:00 2023-11-16T13:56:51.806 INFO:teuthology.orchestra.run.smithi112.stdout:(2/23): ceph-selinux-16.2.14-180.g9513c28f.el8. 514 kB/s | 26 kB 00:00 2023-11-16T13:56:51.815 INFO:teuthology.orchestra.run.smithi061.stdout:(7/23): python3-ceph-common-16.2.14-180.g9513c2 1.7 MB/s | 99 kB 00:00 2023-11-16T13:56:51.907 INFO:teuthology.orchestra.run.smithi061.stdout:(8/23): python3-cephfs-16.2.14-180.g9513c28f.el 2.3 MB/s | 215 kB 00:00 2023-11-16T13:56:51.923 INFO:teuthology.orchestra.run.smithi112.stdout:(3/23): libcephfs2-16.2.14-180.g9513c28f.el8.x8 6.9 MB/s | 824 kB 00:00 2023-11-16T13:56:51.999 INFO:teuthology.orchestra.run.smithi061.stdout:(9/23): python3-rados-16.2.14-180.g9513c28f.el8 4.2 MB/s | 390 kB 00:00 2023-11-16T13:56:52.006 INFO:teuthology.orchestra.run.smithi112.stdout:(4/23): libradosstriper1-16.2.14-180.g9513c28f. 5.8 MB/s | 494 kB 00:00 2023-11-16T13:56:52.058 INFO:teuthology.orchestra.run.smithi061.stdout:(10/23): python3-rbd-16.2.14-180.g9513c28f.el8. 6.2 MB/s | 369 kB 00:00 2023-11-16T13:56:52.133 INFO:teuthology.orchestra.run.smithi061.stdout:(11/23): python3-rgw-16.2.14-180.g9513c28f.el8. 1.5 MB/s | 114 kB 00:00 2023-11-16T13:56:52.147 INFO:teuthology.orchestra.run.smithi112.stdout:(5/23): ceph-base-16.2.14-180.g9513c28f.el8.x86 2.2 MB/s | 6.1 MB 00:02 2023-11-16T13:56:52.265 INFO:teuthology.orchestra.run.smithi112.stdout:(6/23): python3-ceph-argparse-16.2.14-180.g9513 382 kB/s | 45 kB 00:00 2023-11-16T13:56:52.324 INFO:teuthology.orchestra.run.smithi112.stdout:(7/23): librgw2-16.2.14-180.g9513c28f.el8.x86_6 12 MB/s | 3.8 MB 00:00 2023-11-16T13:56:52.349 INFO:teuthology.orchestra.run.smithi112.stdout:(8/23): python3-ceph-common-16.2.14-180.g9513c2 1.2 MB/s | 99 kB 00:00 2023-11-16T13:56:52.374 INFO:teuthology.orchestra.run.smithi112.stdout:(9/23): python3-cephfs-16.2.14-180.g9513c28f.el 4.2 MB/s | 215 kB 00:00 2023-11-16T13:56:52.441 INFO:teuthology.orchestra.run.smithi112.stdout:(10/23): python3-rbd-16.2.14-180.g9513c28f.el8. 5.4 MB/s | 369 kB 00:00 2023-11-16T13:56:52.483 INFO:teuthology.orchestra.run.smithi112.stdout:(11/23): python3-rgw-16.2.14-180.g9513c28f.el8. 2.7 MB/s | 114 kB 00:00 2023-11-16T13:56:52.509 INFO:teuthology.orchestra.run.smithi112.stdout:(12/23): python3-rados-16.2.14-180.g9513c28f.el 2.4 MB/s | 390 kB 00:00 2023-11-16T13:56:52.634 INFO:teuthology.orchestra.run.smithi061.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 708 kB/s | 354 kB 00:00 2023-11-16T13:56:52.767 INFO:teuthology.orchestra.run.smithi061.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2023-11-16T13:56:52.976 INFO:teuthology.orchestra.run.smithi112.stdout:(13/23): ceph-radosgw-16.2.14-180.g9513c28f.el8 3.2 MB/s | 12 MB 00:03 2023-11-16T13:56:53.001 INFO:teuthology.orchestra.run.smithi061.stdout:(14/23): ceph-radosgw-16.2.14-180.g9513c28f.el8 3.3 MB/s | 12 MB 00:03 2023-11-16T13:56:53.001 INFO:teuthology.orchestra.run.smithi112.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 527 kB/s | 259 kB 00:00 2023-11-16T13:56:53.026 INFO:teuthology.orchestra.run.smithi061.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 169 kB/s | 44 kB 00:00 2023-11-16T13:56:53.068 INFO:teuthology.orchestra.run.smithi112.stdout:(15/23): librdkafka-0.11.4-3.el8.x86_64.rpm 606 kB/s | 354 kB 00:00 2023-11-16T13:56:53.177 INFO:teuthology.orchestra.run.smithi112.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2023-11-16T13:56:53.201 INFO:teuthology.orchestra.run.smithi061.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2023-11-16T13:56:53.210 INFO:teuthology.orchestra.run.smithi112.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 188 kB/s | 44 kB 00:00 2023-11-16T13:56:53.302 INFO:teuthology.orchestra.run.smithi061.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-11-16T13:56:53.369 INFO:teuthology.orchestra.run.smithi061.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 885 kB/s | 59 kB 00:00 2023-11-16T13:56:53.419 INFO:teuthology.orchestra.run.smithi112.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2023-11-16T13:56:53.444 INFO:teuthology.orchestra.run.smithi061.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 747 kB/s | 181 kB 00:00 2023-11-16T13:56:53.469 INFO:teuthology.orchestra.run.smithi112.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 619 kB/s | 181 kB 00:00 2023-11-16T13:56:53.470 INFO:teuthology.orchestra.run.smithi061.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 749 kB/s | 75 kB 00:00 2023-11-16T13:56:53.494 INFO:teuthology.orchestra.run.smithi112.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 719 kB/s | 306 kB 00:00 2023-11-16T13:56:53.536 INFO:teuthology.orchestra.run.smithi112.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 640 kB/s | 75 kB 00:00 2023-11-16T13:56:54.153 INFO:teuthology.orchestra.run.smithi061.stdout:(21/23): librados2-16.2.14-180.g9513c28f.el8.x8 5.3 MB/s | 3.7 MB 00:00 2023-11-16T13:56:54.278 INFO:teuthology.orchestra.run.smithi061.stdout:(22/23): librbd1-16.2.14-180.g9513c28f.el8.x86_ 5.2 MB/s | 4.2 MB 00:00 2023-11-16T13:56:54.278 INFO:teuthology.orchestra.run.smithi112.stdout:(22/23): librbd1-16.2.14-180.g9513c28f.el8.x86_ 5.3 MB/s | 4.2 MB 00:00 2023-11-16T13:56:54.586 INFO:teuthology.orchestra.run.smithi112.stdout:(23/23): librados2-16.2.14-180.g9513c28f.el8.x8 3.4 MB/s | 3.7 MB 00:01 2023-11-16T13:56:54.591 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:56:54.591 INFO:teuthology.orchestra.run.smithi112.stdout:Total 10 MB/s | 57 MB 00:05 2023-11-16T13:56:54.631 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:56:54.678 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:56:54.678 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:56:54.979 INFO:teuthology.orchestra.run.smithi061.stdout:(23/23): ceph-common-16.2.14-180.g9513c28f.el8. 4.4 MB/s | 24 MB 00:05 2023-11-16T13:56:54.981 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:56:54.982 INFO:teuthology.orchestra.run.smithi061.stdout:Total 10 MB/s | 57 MB 00:05 2023-11-16T13:56:55.024 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:56:55.025 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:56:55.028 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:56:55.076 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:56:55.076 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:56:55.433 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:56:55.436 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:56:56.222 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:56:56.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-11-16T13:56:56.452 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:56:56.616 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-16T13:56:56.645 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-11-16T13:56:56.792 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-16T13:56:56.999 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-16T13:56:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:56:57.202 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-16T13:56:57.269 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:56:57.367 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librados2-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:56:57.488 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:56:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 4/25 2023-11-16T13:56:57.720 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 4/25 2023-11-16T13:56:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:56:57.917 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:56:58.145 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:56:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 6/25 2023-11-16T13:56:58.355 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:56:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 6/25 2023-11-16T13:56:58.526 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 6/25 2023-11-16T13:56:58.589 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-11-16T13:56:58.591 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 6/25 2023-11-16T13:56:58.724 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-16T13:56:58.759 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-11-16T13:56:58.934 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-16T13:56:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-16T13:56:59.257 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 9/25 2023-11-16T13:56:59.330 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-16T13:56:59.340 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 9/25 2023-11-16T13:56:59.459 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 9/25 2023-11-16T13:56:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-11-16T13:56:59.540 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 9/25 2023-11-16T13:56:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8. 11/25 2023-11-16T13:56:59.706 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-11-16T13:56:59.882 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8. 11/25 2023-11-16T13:56:59.893 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 12/25 2023-11-16T13:57:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 12/25 2023-11-16T13:57:00.125 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 13/25 2023-11-16T13:57:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 14/25 2023-11-16T13:57:00.345 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 13/25 2023-11-16T13:57:00.465 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 15/25 2023-11-16T13:57:00.536 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 15/25 2023-11-16T13:57:00.545 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 14/25 2023-11-16T13:57:00.692 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 15/25 2023-11-16T13:57:00.786 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 15/25 2023-11-16T13:57:00.840 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-16T13:57:01.003 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-16T13:57:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-11-16T13:57:01.167 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-11-16T13:57:01.213 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-11-16T13:57:01.349 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-11-16T13:57:01.407 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x8 19/25 2023-11-16T13:57:01.534 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x8 19/25 2023-11-16T13:57:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:04.157 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:04.175 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:04.849 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:05.131 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 21/25 2023-11-16T13:57:05.160 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 20/25 2023-11-16T13:57:05.262 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 21/25 2023-11-16T13:57:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:05.635 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 21/25 2023-11-16T13:57:05.752 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 21/25 2023-11-16T13:57:05.875 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:23.496 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:23.496 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-11-16T13:57:23.497 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:24.704 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 23/25 2023-11-16T13:57:24.775 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 23/25 2023-11-16T13:57:24.775 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /sys 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /proc 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /run 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /mnt 2023-11-16T13:57:24.788 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /var/tmp 2023-11-16T13:57:24.789 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /home 2023-11-16T13:57:24.789 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /tmp 2023-11-16T13:57:24.789 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-11-16T13:57:24.789 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:24.981 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-16T13:57:24.981 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:25.665 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 1/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 4/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 6/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 7/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8. 8/25 2023-11-16T13:57:25.666 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x8 9/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 10/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 11/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 12/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 13/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-11-16T13:57:25.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-11-16T13:57:25.668 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:25.668 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-16T13:57:25.668 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 24/25 2023-11-16T13:57:25.999 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 23/25 2023-11-16T13:57:26.076 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 23/25 2023-11-16T13:57:26.076 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-16T13:57:26.275 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-16T13:57:26.275 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T13:57:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb-1.22-1.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T13:57:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:26.732 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:57:26.942 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 1/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 3/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 4/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 5/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_6 6/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 7/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8. 8/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x8 9/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 10/25 2023-11-16T13:57:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 11/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 12/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 13/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-2:16.2.14-180.g9513c28f.el8.x86_64 22/25 2023-11-16T13:57:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-16T13:57:26.945 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 24/25 2023-11-16T13:57:26.968 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2023-11-16T13:57:27.816 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:43 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:57:28.265 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-16T13:57:28.265 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:28.265 INFO:teuthology.orchestra.run.smithi061.stdout:Upgraded: 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: librados2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: leveldb-1.22-1.el8.x86_64 2023-11-16T13:57:28.266 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.267 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T13:57:28.268 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.268 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.268 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:28.268 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:28.268 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:57:28.541 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-test 2023-11-16T13:57:29.423 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:45 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:57:29.487 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:57:29.488 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:29.488 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:57:29.488 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:29.488 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:16.2.14-180.g9513c28f.el8 ceph 52 M 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:16.2.14-180.g9513c28f.el8 ceph 174 k 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-16T13:57:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-16T13:57:29.490 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-16T13:57:29.490 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:29.490 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:57:29.490 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:29.490 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-11-16T13:57:29.491 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:29.491 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 53 M 2023-11-16T13:57:29.491 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 196 M 2023-11-16T13:57:29.492 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:57:30.057 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 537 kB/s | 203 kB 00:00 2023-11-16T13:57:30.082 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): libcephsqlite-16.2.14-180.g9513c28f.el8. 430 kB/s | 174 kB 00:00 2023-11-16T13:57:30.191 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-11-16T13:57:30.299 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 653 kB/s | 71 kB 00:00 2023-11-16T13:57:30.591 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 635 kB/s | 323 kB 00:00 2023-11-16T13:57:31.191 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:57:31.192 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:31.192 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:57:31.192 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:31.192 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test x86_64 2:16.2.14-180.g9513c28f.el8 ceph 52 M 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite x86_64 2:16.2.14-180.g9513c28f.el8 ceph 174 k 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:31.193 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-11-16T13:57:31.194 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:31.194 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 53 M 2023-11-16T13:57:31.194 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 196 M 2023-11-16T13:57:31.194 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:57:31.577 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 591 kB/s | 203 kB 00:00 2023-11-16T13:57:31.674 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): libcephsqlite-16.2.14-180.g9513c28f.el8. 394 kB/s | 174 kB 00:00 2023-11-16T13:57:31.691 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-11-16T13:57:31.775 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 846 kB/s | 71 kB 00:00 2023-11-16T13:57:32.042 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 880 kB/s | 323 kB 00:00 2023-11-16T13:57:32.583 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-test-16.2.14-180.g9513c28f.el8.x86_ 18 MB/s | 52 MB 00:02 2023-11-16T13:57:32.585 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:57:32.585 INFO:teuthology.orchestra.run.smithi112.stdout:Total 17 MB/s | 53 MB 00:03 2023-11-16T13:57:32.628 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:57:32.648 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:57:32.648 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:57:32.934 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:57:32.937 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:57:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:57:33.689 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-16T13:57:33.898 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-16T13:57:34.028 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T13:57:34.095 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T13:57:34.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-16T13:57:34.469 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 5/6 2023-11-16T13:57:34.892 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): ceph-test-16.2.14-180.g9513c28f.el8.x86_ 14 MB/s | 52 MB 00:03 2023-11-16T13:57:34.894 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:57:34.894 INFO:teuthology.orchestra.run.smithi061.stdout:Total 14 MB/s | 53 MB 00:03 2023-11-16T13:57:34.939 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:57:34.961 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:57:34.961 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:57:35.263 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:57:35.266 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:57:35.900 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:57:36.136 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-16T13:57:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-16T13:57:36.463 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T13:57:36.547 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T13:57:36.736 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-16T13:57:36.912 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 5/6 2023-11-16T13:57:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 5/6 2023-11-16T13:57:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 6/6 2023-11-16T13:57:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 6/6 2023-11-16T13:57:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 1/6 2023-11-16T13:57:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 2/6 2023-11-16T13:57:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-16T13:57:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-16T13:57:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-16T13:57:40.276 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-16T13:57:40.276 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: jq-1.6-8.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:40.277 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:57:40.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2023-11-16T13:57:41.376 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:57 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:57:41.782 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 5/6 2023-11-16T13:57:42.036 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 6/6 2023-11-16T13:57:42.329 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 6/6 2023-11-16T13:57:42.329 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 1/6 2023-11-16T13:57:42.329 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 2/6 2023-11-16T13:57:42.329 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-16T13:57:42.330 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-16T13:57:42.330 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-16T13:57:43.030 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:57:43.033 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:43.033 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:57:43.033 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:16.2.14-180.g9513c28f.el8 ceph 6.4 k 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:16.2.14-180.g9513c28f.el8 ceph 2.4 M 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:16.2.14-180.g9513c28f.el8 ceph 1.6 M 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 245 k 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:16.2.14-180.g9513c28f.el8 ceph 5.0 M 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 18 M 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T13:57:43.034 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T13:57:43.035 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T13:57:43.036 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout:Install 32 Packages 2023-11-16T13:57:43.037 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:43.038 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 31 M 2023-11-16T13:57:43.038 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 101 M 2023-11-16T13:57:43.038 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: jq-1.6-8.el8.x86_64 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T13:57:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T13:57:43.082 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:43.082 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:57:43.295 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph 2023-11-16T13:57:43.501 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-16.2.14-180.g9513c28f.el8.x86_64.r 29 kB/s | 6.4 kB 00:00 2023-11-16T13:57:43.901 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-mgr-16.2.14-180.g9513c28f.el8.x86_ 2.6 MB/s | 1.6 MB 00:00 2023-11-16T13:57:44.026 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-mds-16.2.14-180.g9513c28f.el8.x86_ 3.3 MB/s | 2.4 MB 00:00 2023-11-16T13:57:44.093 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): ceph-mgr-modules-core-16.2.14-180.g9513 3.6 MB/s | 245 kB 00:00 2023-11-16T13:57:44.191 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:00 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:57:44.577 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 325 kB/s | 157 kB 00:00 2023-11-16T13:57:44.702 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 311 kB/s | 39 kB 00:00 2023-11-16T13:57:44.861 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 653 kB/s | 103 kB 00:00 2023-11-16T13:57:44.995 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 428 kB/s | 57 kB 00:00 2023-11-16T13:57:45.237 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): ceph-osd-16.2.14-180.g9513c28f.el8.x86_ 14 MB/s | 18 MB 00:01 2023-11-16T13:57:45.270 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2023-11-16T13:57:45.620 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 649 kB/s | 249 kB 00:00 2023-11-16T13:57:45.729 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2023-11-16T13:57:45.763 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.3 MB/s | 641 kB 00:00 2023-11-16T13:57:45.978 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:57:45.982 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:45.982 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:57:45.982 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:45.982 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:57:45.982 INFO:teuthology.orchestra.run.smithi061.stdout: ceph x86_64 2:16.2.14-180.g9513c28f.el8 ceph 6.4 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds x86_64 2:16.2.14-180.g9513c28f.el8 ceph 2.4 M 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr x86_64 2:16.2.14-180.g9513c28f.el8 ceph 1.6 M 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 245 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon x86_64 2:16.2.14-180.g9513c28f.el8 ceph 5.0 M 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 18 M 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T13:57:45.983 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T13:57:45.984 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout:Install 32 Packages 2023-11-16T13:57:45.985 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:57:45.986 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 31 M 2023-11-16T13:57:45.986 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 101 M 2023-11-16T13:57:45.986 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:57:46.096 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 119 kB/s | 44 kB 00:00 2023-11-16T13:57:46.172 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): ceph-mon-16.2.14-180.g9513c28f.el8.x86 1.9 MB/s | 5.0 MB 00:02 2023-11-16T13:57:46.272 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 364 kB/s | 185 kB 00:00 2023-11-16T13:57:46.364 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 647 kB/s | 173 kB 00:00 2023-11-16T13:57:46.389 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-11-16T13:57:46.456 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2023-11-16T13:57:46.498 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2023-11-16T13:57:46.560 INFO:teuthology.orchestra.run.smithi061.stdout:(1/32): ceph-16.2.14-180.g9513c28f.el8.x86_64.r 24 kB/s | 6.4 kB 00:00 2023-11-16T13:57:46.565 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 541 kB/s | 59 kB 00:00 2023-11-16T13:57:46.660 INFO:teuthology.orchestra.run.smithi061.stdout:(2/32): ceph-mgr-16.2.14-180.g9513c28f.el8.x86_ 4.4 MB/s | 1.6 MB 00:00 2023-11-16T13:57:46.665 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-11-16T13:57:46.690 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-11-16T13:57:46.782 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2023-11-16T13:57:46.799 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2023-11-16T13:57:46.858 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): python3-cherrypy-18.4.0-1.el8.noarch.r 561 kB/s | 384 kB 00:00 2023-11-16T13:57:46.891 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2023-11-16T13:57:46.933 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2023-11-16T13:57:46.992 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2023-11-16T13:57:47.034 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 852 kB/s | 85 kB 00:00 2023-11-16T13:57:47.059 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-11-16T13:57:47.093 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-11-16T13:57:47.096 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:57:47.097 INFO:teuthology.orchestra.run.smithi112.stdout:Total 7.7 MB/s | 31 MB 00:04 2023-11-16T13:57:47.181 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:57:47.222 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:57:47.222 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:57:47.551 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:57:47.554 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:57:47.735 INFO:teuthology.orchestra.run.smithi061.stdout:(3/32): ceph-osd-16.2.14-180.g9513c28f.el8.x86_ 17 MB/s | 18 MB 00:01 2023-11-16T13:57:47.785 INFO:teuthology.orchestra.run.smithi061.stdout:(4/32): ceph-mgr-modules-core-16.2.14-180.g9513 4.8 MB/s | 245 kB 00:00 2023-11-16T13:57:48.036 INFO:teuthology.orchestra.run.smithi061.stdout:(5/32): ceph-mds-16.2.14-180.g9513c28f.el8.x86_ 1.4 MB/s | 2.4 MB 00:01 2023-11-16T13:57:48.094 INFO:teuthology.orchestra.run.smithi061.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 509 kB/s | 157 kB 00:00 2023-11-16T13:57:48.194 INFO:teuthology.orchestra.run.smithi061.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2023-11-16T13:57:48.270 INFO:teuthology.orchestra.run.smithi061.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 166 kB/s | 39 kB 00:00 2023-11-16T13:57:48.295 INFO:teuthology.orchestra.run.smithi061.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 568 kB/s | 57 kB 00:00 2023-11-16T13:57:48.579 INFO:teuthology.orchestra.run.smithi061.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 878 kB/s | 249 kB 00:00 2023-11-16T13:57:48.604 INFO:teuthology.orchestra.run.smithi061.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2023-11-16T13:57:48.650 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:57:48.713 INFO:teuthology.orchestra.run.smithi061.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 4.7 MB/s | 641 kB 00:00 2023-11-16T13:57:48.829 INFO:teuthology.orchestra.run.smithi061.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2023-11-16T13:57:48.847 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-16T13:57:48.889 INFO:teuthology.orchestra.run.smithi061.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 248 kB/s | 44 kB 00:00 2023-11-16T13:57:49.022 INFO:teuthology.orchestra.run.smithi061.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-11-16T13:57:49.072 INFO:teuthology.orchestra.run.smithi061.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 764 kB/s | 185 kB 00:00 2023-11-16T13:57:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-16T13:57:49.114 INFO:teuthology.orchestra.run.smithi061.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2023-11-16T13:57:49.140 INFO:teuthology.orchestra.run.smithi061.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2023-11-16T13:57:49.173 INFO:teuthology.orchestra.run.smithi061.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 300 kB/s | 18 kB 00:00 2023-11-16T13:57:49.199 INFO:teuthology.orchestra.run.smithi061.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 831 kB/s | 49 kB 00:00 2023-11-16T13:57:49.232 INFO:teuthology.orchestra.run.smithi061.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2023-11-16T13:57:49.258 INFO:teuthology.orchestra.run.smithi061.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-11-16T13:57:49.283 INFO:teuthology.orchestra.run.smithi061.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-16T13:57:49.341 INFO:teuthology.orchestra.run.smithi061.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2023-11-16T13:57:49.366 INFO:teuthology.orchestra.run.smithi061.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 286 kB/s | 24 kB 00:00 2023-11-16T13:57:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-16T13:57:49.392 INFO:teuthology.orchestra.run.smithi061.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2023-11-16T13:57:49.442 INFO:teuthology.orchestra.run.smithi061.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-11-16T13:57:49.484 INFO:teuthology.orchestra.run.smithi061.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-11-16T13:57:49.543 INFO:teuthology.orchestra.run.smithi061.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2023-11-16T13:57:49.568 INFO:teuthology.orchestra.run.smithi061.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2023-11-16T13:57:49.590 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-16T13:57:49.593 INFO:teuthology.orchestra.run.smithi061.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2023-11-16T13:57:49.771 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-16T13:57:49.893 INFO:teuthology.orchestra.run.smithi061.stdout:(32/32): ceph-mon-16.2.14-180.g9513c28f.el8.x86 1.5 MB/s | 5.0 MB 00:03 2023-11-16T13:57:49.897 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:57:49.897 INFO:teuthology.orchestra.run.smithi061.stdout:Total 8.0 MB/s | 31 MB 00:03 2023-11-16T13:57:49.960 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-16T13:57:49.984 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:57:50.027 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:57:50.027 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:57:50.178 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-16T13:57:50.340 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:57:50.343 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:57:50.356 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-16T13:57:50.565 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-16T13:57:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-16T13:57:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-16T13:57:51.289 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-16T13:57:51.588 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:57:51.693 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-16T13:57:51.812 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-16T13:57:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-16T13:57:52.069 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-16T13:57:52.150 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-16T13:57:52.350 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-16T13:57:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-16T13:57:52.575 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-16T13:57:52.693 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T13:57:52.757 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-16T13:57:52.947 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-16T13:57:53.018 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-16T13:57:53.114 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-16T13:57:53.247 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-16T13:57:53.292 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-16T13:57:53.419 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-16T13:57:53.504 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-16T13:57:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-16T13:57:53.732 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-16T13:57:53.855 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-16T13:57:53.987 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-16T13:57:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-16T13:57:54.228 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-16T13:57:54.264 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-16T13:57:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-16T13:57:54.648 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-16T13:57:54.906 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-16T13:57:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-16T13:57:55.085 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-16T13:57:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8. 27/32 2023-11-16T13:57:55.361 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-16T13:57:55.508 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 28/32 2023-11-16T13:57:55.679 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T13:57:56.031 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-16T13:57:56.276 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-16T13:57:56.440 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-16T13:57:56.624 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-16T13:57:56.911 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-16T13:57:56.999 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 28/32 2023-11-16T13:57:57.124 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-16T13:57:57.166 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 29/32 2023-11-16T13:57:57.402 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-16T13:57:57.592 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 29/32 2023-11-16T13:57:57.734 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-16T13:57:57.767 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 30/32 2023-11-16T13:57:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 30/32 2023-11-16T13:57:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-16T13:57:58.141 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 31/32 2023-11-16T13:57:58.159 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 31/32 2023-11-16T13:57:58.299 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:16.2.14-180.g9513c28f.el8.x86_64 32/32 2023-11-16T13:57:58.447 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8. 27/32 2023-11-16T13:57:58.695 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 28/32 2023-11-16T13:57:58.707 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:16.2.14-180.g9513c28f.el8.x86_64 32/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:16.2.14-180.g9513c28f.el8.x86_64 1/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 2/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 3/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 4/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 5/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8. 6/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-16T13:57:58.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-16T13:57:58.709 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-16T13:57:58.710 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-16T13:57:58.710 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-16T13:57:58.710 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-16T13:57:58.710 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-16T13:57:58.710 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-16T13:57:58.711 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-16T13:57:58.711 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-16T13:57:58.711 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-16T13:57:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-16T13:57:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:57:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:57:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T13:57:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T13:57:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-16T13:57:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T13:57:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T13:57:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T13:57:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T13:57:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T13:57:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T13:57:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T13:57:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T13:57:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:57:59.922 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:00.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2023-11-16T13:58:00.275 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 28/32 2023-11-16T13:58:00.460 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 29/32 2023-11-16T13:58:00.912 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 29/32 2023-11-16T13:58:01.017 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:17 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:01.127 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 30/32 2023-11-16T13:58:01.355 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 30/32 2023-11-16T13:58:01.568 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 31/32 2023-11-16T13:58:01.587 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 31/32 2023-11-16T13:58:01.710 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-2:16.2.14-180.g9513c28f.el8.x86_64 32/32 2023-11-16T13:58:02.125 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-2:16.2.14-180.g9513c28f.el8.x86_64 32/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-2:16.2.14-180.g9513c28f.el8.x86_64 1/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 2/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 3/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 4/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 5/32 2023-11-16T13:58:02.126 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8. 6/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-16T13:58:02.127 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-16T13:58:02.128 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-16T13:58:02.129 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-16T13:58:02.130 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-16T13:58:02.130 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-16T13:58:02.130 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-16T13:58:02.130 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-16T13:58:02.130 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-16T13:58:02.465 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:58:02.679 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:02.680 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:58:02.681 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:02.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2023-11-16T13:58:03.481 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-16T13:58:03.481 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T13:58:03.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T13:58:03.483 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:03.484 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:03.618 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:19 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:03.626 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-base 2023-11-16T13:58:04.522 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:20 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:05.292 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:05.293 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 85 k 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:05.294 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 85 k 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 352 k 2023-11-16T13:58:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:05.607 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-16.2.14-180.g9513c28f.el8.noarch.rpm 273 kB/s | 85 kB 00:00 2023-11-16T13:58:05.608 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:05.608 INFO:teuthology.orchestra.run.smithi112.stdout:Total 272 kB/s | 85 kB 00:00 2023-11-16T13:58:05.608 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:05.614 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:05.614 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:05.616 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:05.619 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:06.072 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-base-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:58:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:06.304 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:06.305 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:58:06.306 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:06.436 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install cephadm 2023-11-16T13:58:06.578 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:07.225 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:07.326 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:23 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:07.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2023-11-16T13:58:08.327 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:24 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:09.098 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 85 k 2023-11-16T13:58:09.099 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 85 k 2023-11-16T13:58:09.100 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 352 k 2023-11-16T13:58:09.101 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:09.392 INFO:teuthology.orchestra.run.smithi061.stdout:cephadm-16.2.14-180.g9513c28f.el8.noarch.rpm 293 kB/s | 85 kB 00:00 2023-11-16T13:58:09.392 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:09.392 INFO:teuthology.orchestra.run.smithi061.stdout:Total 292 kB/s | 85 kB 00:00 2023-11-16T13:58:09.393 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:58:09.400 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:58:09.400 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:58:09.402 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:58:09.405 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:58:09.452 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:58:09.941 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:10.048 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-180.g9513c28f.el8 ceph 161 k 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:58:10.050 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:10.050 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 161 k 2023-11-16T13:58:10.050 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 470 k 2023-11-16T13:58:10.050 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:10.098 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:10.333 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:10.786 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-16.2.14-180.g9513c2 219 kB/s | 161 kB 00:00 2023-11-16T13:58:10.786 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:10.787 INFO:teuthology.orchestra.run.smithi112.stdout:Total 219 kB/s | 161 kB 00:00 2023-11-16T13:58:10.787 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:10.799 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:10.799 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:10.852 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:10.856 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:10.979 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:11.164 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:11.581 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:11.894 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/1 2023-11-16T13:58:11.894 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:11.894 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:11.894 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:11.895 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:11.895 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:12.217 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:12.218 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:12.218 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:12.218 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:12.218 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:12.218 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:12.372 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-immutable-object-cache 2023-11-16T13:58:12.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2023-11-16T13:58:13.285 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:29 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:13.313 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:29 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:14.718 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:58:14.934 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:14.935 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:58:14.935 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:15.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2023-11-16T13:58:15.093 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:15.094 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:15.094 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-11-16T13:58:15.094 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:15.094 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:15.094 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-180.g9513c28f.el8 ceph 161 k 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:15.095 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 161 k 2023-11-16T13:58:15.096 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 470 k 2023-11-16T13:58:15.096 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:15.281 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-immutable-object-cache-16.2.14-180.g9513c2 867 kB/s | 161 kB 00:00 2023-11-16T13:58:15.281 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:15.281 INFO:teuthology.orchestra.run.smithi061.stdout:Total 861 kB/s | 161 kB 00:00 2023-11-16T13:58:15.282 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:58:15.293 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:58:15.293 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:58:15.344 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:58:15.347 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:58:15.475 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:58:15.658 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:15.904 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:31 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:16.097 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:16.785 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-180.g9513c28f. 1/1 2023-11-16T13:58:16.785 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:16.785 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:16.786 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:58:16.786 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:16.786 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:17.029 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr 2023-11-16T13:58:17.571 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 3.1 M 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 24 k 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 15 k 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-16T13:58:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:17.573 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.4 M 2023-11-16T13:58:17.574 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 22 M 2023-11-16T13:58:17.574 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:17.924 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:33 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:18.148 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-prometheus-alerts-16.2.14-180.g9513 76 kB/s | 15 kB 00:00 2023-11-16T13:58:18.198 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-grafana-dashboards-16.2.14-180.g951 99 kB/s | 24 kB 00:00 2023-11-16T13:58:18.309 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 304 kB/s | 34 kB 00:00 2023-11-16T13:58:18.363 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 198 kB/s | 43 kB 00:00 2023-11-16T13:58:18.427 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2023-11-16T13:58:19.468 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-mgr-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:58:19.577 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-mgr-dashboard-16.2.14-180.g9513c28f 1.9 MB/s | 3.1 MB 00:01 2023-11-16T13:58:19.578 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:19.579 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.7 MB/s | 3.4 MB 00:02 2023-11-16T13:58:19.591 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:19.602 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:19.602 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:19.698 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:19.699 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:58:19.699 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:19.767 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:19.770 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:19.795 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-dashboard 2023-11-16T13:58:20.040 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:20.269 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-16T13:58:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-16T13:58:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-16T13:58:20.678 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:36 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.n 4/6 2023-11-16T13:58:21.982 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8. 5/6 2023-11-16T13:58:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 6/6 2023-11-16T13:58:22.447 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 3.1 M 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:58:22.448 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 24 k 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 15 k 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-11-16T13:58:22.449 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:22.450 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 3.4 M 2023-11-16T13:58:22.450 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 22 M 2023-11-16T13:58:22.450 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 6/6 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8. 1/6 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 2/6 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.n 3/6 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-16T13:58:22.540 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-16T13:58:23.017 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): ceph-grafana-dashboards-16.2.14-180.g951 72 kB/s | 24 kB 00:00 2023-11-16T13:58:23.142 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): ceph-prometheus-alerts-16.2.14-180.g9513 32 kB/s | 15 kB 00:00 2023-11-16T13:58:23.201 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T13:58:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T13:58:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T13:58:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:23.208 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:23.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-16T13:58:23.443 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 112 kB/s | 34 kB 00:00 2023-11-16T13:58:23.777 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 341 kB/s | 196 kB 00:00 2023-11-16T13:58:24.185 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:40 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:25.837 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:25.838 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-11-16T13:58:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:25.839 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-11-16T13:58:25.839 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 8.6 M 2023-11-16T13:58:25.839 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:58:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-16T13:58:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-16T13:58:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-16T13:58:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout:Install 8 Packages 2023-11-16T13:58:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:25.842 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 37 M 2023-11-16T13:58:25.842 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 210 M 2023-11-16T13:58:25.842 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:26.093 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): ceph-mgr-dashboard-16.2.14-180.g9513c28f 937 kB/s | 3.1 MB 00:03 2023-11-16T13:58:26.095 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:26.096 INFO:teuthology.orchestra.run.smithi061.stdout:Total 964 kB/s | 3.4 MB 00:03 2023-11-16T13:58:26.108 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:58:26.120 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:58:26.120 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:58:26.281 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:58:26.284 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:58:26.558 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:58:26.780 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-16T13:58:27.008 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-16T13:58:27.299 INFO:teuthology.orchestra.run.smithi112.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 4.4 MB/s | 4.7 MB 00:01 2023-11-16T13:58:27.300 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-16T13:58:27.445 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.n 4/6 2023-11-16T13:58:27.533 INFO:teuthology.orchestra.run.smithi112.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.7 MB/s | 4.9 MB 00:01 2023-11-16T13:58:27.608 INFO:teuthology.orchestra.run.smithi112.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2023-11-16T13:58:27.634 INFO:teuthology.orchestra.run.smithi112.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-11-16T13:58:28.118 INFO:teuthology.orchestra.run.smithi112.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 27 MB/s | 14 MB 00:00 2023-11-16T13:58:28.143 INFO:teuthology.orchestra.run.smithi112.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2023-11-16T13:58:28.326 INFO:teuthology.orchestra.run.smithi112.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 828 kB/s | 172 kB 00:00 2023-11-16T13:58:28.593 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8. 5/6 2023-11-16T13:58:28.973 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 6/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 6/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8. 1/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarc 2/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.n 3/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-16T13:58:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-16T13:58:29.976 INFO:teuthology.orchestra.run.smithi112.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-18 2.3 MB/s | 8.6 MB 00:03 2023-11-16T13:58:29.977 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:29.977 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.8 MB/s | 37 MB 00:04 2023-11-16T13:58:29.977 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:29.978 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:30.100 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-16T13:58:30.212 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:30.230 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:30.230 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:30.542 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:30.545 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:30.982 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:46 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T13:58:31.454 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T13:58:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T13:58:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T13:58:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:32.764 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================================ 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================================ 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 8.6 M 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-16T13:58:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================================ 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout:Install 8 Packages 2023-11-16T13:58:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:32.768 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 37 M 2023-11-16T13:58:32.768 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 210 M 2023-11-16T13:58:32.768 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:33.017 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:33.194 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T13:58:33.633 INFO:teuthology.orchestra.run.smithi061.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 7.4 MB/s | 4.7 MB 00:00 2023-11-16T13:58:33.700 INFO:teuthology.orchestra.run.smithi061.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.9 MB/s | 4.9 MB 00:00 2023-11-16T13:58:33.792 INFO:teuthology.orchestra.run.smithi061.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-16T13:58:33.867 INFO:teuthology.orchestra.run.smithi061.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2023-11-16T13:58:33.884 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T13:58:34.238 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T13:58:34.401 INFO:teuthology.orchestra.run.smithi061.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 22 MB/s | 14 MB 00:00 2023-11-16T13:58:34.468 INFO:teuthology.orchestra.run.smithi061.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-16T13:58:34.477 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T13:58:34.652 INFO:teuthology.orchestra.run.smithi061.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 690 kB/s | 172 kB 00:00 2023-11-16T13:58:36.223 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T13:58:36.368 INFO:teuthology.orchestra.run.smithi061.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-18 2.5 MB/s | 8.6 MB 00:03 2023-11-16T13:58:36.368 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:36.369 INFO:teuthology.orchestra.run.smithi061.stdout:Total 10 MB/s | 37 MB 00:03 2023-11-16T13:58:36.609 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:58:36.628 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:58:36.628 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:58:36.959 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:58:36.962 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:58:37.535 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-16T13:58:37.570 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:58:37.729 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T13:58:37.736 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 8/8 2023-11-16T13:58:37.851 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T13:58:38.046 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 8/8 2023-11-16T13:58:38.046 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 1/8 2023-11-16T13:58:38.046 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-16T13:58:38.047 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T13:58:38.662 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T13:58:38.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-16T13:58:38.778 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:38.778 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:38.779 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:38.821 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:38.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2023-11-16T13:58:39.474 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:39.612 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T13:58:39.739 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:55 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:40.347 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T13:58:40.671 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T13:58:40.912 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T13:58:41.406 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:41.407 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:41.407 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 27 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-16T13:58:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2023-11-16T13:58:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:41.410 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.9 M 2023-11-16T13:58:41.410 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2023-11-16T13:58:41.410 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:42.498 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 95 kB/s | 20 kB 00:00 2023-11-16T13:58:42.523 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 113 kB/s | 27 kB 00:00 2023-11-16T13:58:42.557 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): ceph-mgr-rook-16.2.14-180.g9513c28f.el8 102 kB/s | 27 kB 00:00 2023-11-16T13:58:42.690 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 659 kB/s | 126 kB 00:00 2023-11-16T13:58:42.724 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 548 kB/s | 110 kB 00:00 2023-11-16T13:58:42.750 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 805 kB/s | 155 kB 00:00 2023-11-16T13:58:42.790 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T13:58:42.842 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 282 kB/s | 43 kB 00:00 2023-11-16T13:58:43.000 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 120 kB/s | 33 kB 00:00 2023-11-16T13:58:43.025 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 59 kB/s | 16 kB 00:00 2023-11-16T13:58:43.251 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 199 kB/s | 81 kB 00:00 2023-11-16T13:58:43.268 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 265 kB/s | 64 kB 00:00 2023-11-16T13:58:43.393 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 432 kB/s | 61 kB 00:00 2023-11-16T13:58:43.493 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2023-11-16T13:58:43.494 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:43.494 INFO:teuthology.orchestra.run.smithi112.stdout:Total 939 kB/s | 1.9 MB 00:02 2023-11-16T13:58:43.553 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:43.569 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:43.569 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:43.655 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:43.658 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:44.164 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-16T13:58:44.190 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:44.346 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 8/8 2023-11-16T13:58:44.404 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-16T13:58:44.585 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-16T13:58:44.670 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 8/8 2023-11-16T13:58:44.670 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28 1/8 2023-11-16T13:58:44.670 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-16T13:58:44.670 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-16T13:58:44.670 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-16T13:58:44.671 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T13:58:44.671 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-16T13:58:44.671 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-16T13:58:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-16T13:58:45.100 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-16T13:58:45.317 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-16T13:58:45.408 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-16T13:58:45.409 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T13:58:45.409 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T13:58:45.409 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-16T13:58:45.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:45.409 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:45.523 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-rook 2023-11-16T13:58:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-16T13:58:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-16T13:58:46.082 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-16T13:58:46.432 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:02 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:47.109 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-16T13:58:47.374 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-16T13:58:47.599 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-16T13:58:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-16T13:58:47.950 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 13/13 2023-11-16T13:58:48.248 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 27 k 2023-11-16T13:58:48.250 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-16T13:58:48.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout:Install 13 Packages 2023-11-16T13:58:48.252 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:48.253 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 1.9 M 2023-11-16T13:58:48.253 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 24 M 2023-11-16T13:58:48.253 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 13/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 1/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-16T13:58:48.255 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-16T13:58:48.256 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-16T13:58:48.256 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-16T13:58:48.256 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-16T13:58:48.256 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-16T13:58:48.916 INFO:teuthology.orchestra.run.smithi061.stdout:(1/13): ceph-mgr-rook-16.2.14-180.g9513c28f.el8 189 kB/s | 27 kB 00:00 2023-11-16T13:58:49.017 INFO:teuthology.orchestra.run.smithi061.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 81 kB/s | 20 kB 00:00 2023-11-16T13:58:49.050 INFO:teuthology.orchestra.run.smithi061.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 95 kB/s | 27 kB 00:00 2023-11-16T13:58:49.162 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-16T13:58:49.162 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:49.162 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T13:58:49.163 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T13:58:49.164 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:49.164 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:49.281 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2023-11-16T13:58:49.317 INFO:teuthology.orchestra.run.smithi061.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 366 kB/s | 110 kB 00:00 2023-11-16T13:58:49.401 INFO:teuthology.orchestra.run.smithi061.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 261 kB/s | 126 kB 00:00 2023-11-16T13:58:49.485 INFO:teuthology.orchestra.run.smithi061.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 357 kB/s | 155 kB 00:00 2023-11-16T13:58:49.635 INFO:teuthology.orchestra.run.smithi061.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2023-11-16T13:58:49.685 INFO:teuthology.orchestra.run.smithi061.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 116 kB/s | 33 kB 00:00 2023-11-16T13:58:49.769 INFO:teuthology.orchestra.run.smithi061.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 57 kB/s | 16 kB 00:00 2023-11-16T13:58:49.902 INFO:teuthology.orchestra.run.smithi061.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 484 kB/s | 64 kB 00:00 2023-11-16T13:58:49.961 INFO:teuthology.orchestra.run.smithi061.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 250 kB/s | 81 kB 00:00 2023-11-16T13:58:49.986 INFO:teuthology.orchestra.run.smithi061.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2023-11-16T13:58:50.020 INFO:teuthology.orchestra.run.smithi061.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.5 MB/s | 1.2 MB 00:00 2023-11-16T13:58:50.023 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:50.023 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-11-16T13:58:50.085 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:58:50.102 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:58:50.102 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:58:50.140 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:06 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:50.196 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:58:50.199 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:58:50.776 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:58:51.008 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-16T13:58:51.196 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-16T13:58:51.423 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-16T13:58:51.658 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-16T13:58:51.807 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:51.808 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:51.808 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:51.808 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 116 k 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-11-16T13:58:51.809 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:51.810 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 5.8 M 2023-11-16T13:58:51.810 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2023-11-16T13:58:51.810 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:51.860 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-16T13:58:52.085 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-16T13:58:52.379 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-16T13:58:52.542 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-16.2.14-180.g9513c28f.e 254 kB/s | 116 kB 00:00 2023-11-16T13:58:52.642 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 965 kB/s | 538 kB 00:00 2023-11-16T13:58:52.642 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-16T13:58:52.959 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.5 MB/s | 4.8 MB 00:00 2023-11-16T13:58:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 48 kB/s | 21 kB 00:00 2023-11-16T13:58:53.478 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 427 kB/s | 355 kB 00:00 2023-11-16T13:58:53.503 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 90 kB/s | 49 kB 00:00 2023-11-16T13:58:53.506 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:58:53.507 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.4 MB/s | 5.8 MB 00:01 2023-11-16T13:58:53.561 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:58:53.571 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:58:53.571 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:58:53.679 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:58:53.682 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:58:53.732 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-16T13:58:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:58:54.216 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-16T13:58:54.301 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-16T13:58:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-16T13:58:54.519 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-16T13:58:54.703 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-16T13:58:54.844 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 13/13 2023-11-16T13:58:55.151 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 13/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 1/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-16T13:58:55.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-16T13:58:55.153 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-16T13:58:55.153 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-16T13:58:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-16T13:58:55.828 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-16T13:58:55.942 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T13:58:55.943 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:55.944 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:58:56.049 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-16T13:58:56.057 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-cephadm 2023-11-16T13:58:56.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 6/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 6/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-16T13:58:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-16T13:58:56.957 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:12 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: python3-apipkg-1.5-6.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-11-16T13:58:57.248 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:57.249 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:58:57.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2023-11-16T13:58:58.219 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:14 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:58:58.732 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm noarch 2:16.2.14-180.g9513c28f.el8 ceph-noarch 116 k 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:58:58.733 INFO:teuthology.orchestra.run.smithi061.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-11-16T13:58:58.734 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:58:58.735 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 5.8 M 2023-11-16T13:58:58.735 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 24 M 2023-11-16T13:58:58.735 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:58:59.679 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): ceph-mgr-cephadm-16.2.14-180.g9513c28f.e 285 kB/s | 116 kB 00:00 2023-11-16T13:58:59.721 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2023-11-16T13:58:59.858 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:58:59.862 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-11-16T13:58:59.862 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:59.862 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:58:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 910 k 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:58:59.863 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2023-11-16T13:58:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:58:59.864 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 993 k 2023-11-16T13:58:59.864 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.8 M 2023-11-16T13:58:59.864 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:58:59.988 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 69 kB/s | 21 kB 00:00 2023-11-16T13:59:00.055 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.1 MB/s | 4.8 MB 00:00 2023-11-16T13:59:00.095 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 419 kB/s | 83 kB 00:00 2023-11-16T13:59:00.238 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 195 kB/s | 49 kB 00:00 2023-11-16T13:59:00.254 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-16.2.14-180.g9513c28f.el8.x86_ 2.5 MB/s | 910 kB 00:00 2023-11-16T13:59:00.254 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:00.255 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.5 MB/s | 993 kB 00:00 2023-11-16T13:59:00.265 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:00.286 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:00.286 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:00.371 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:00.374 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:00.380 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 539 kB/s | 355 kB 00:00 2023-11-16T13:59:00.382 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:00.383 INFO:teuthology.orchestra.run.smithi061.stdout:Total 3.5 MB/s | 5.8 MB 00:01 2023-11-16T13:59:00.438 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:00.449 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:00.449 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:00.556 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:00.560 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:00.566 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-16T13:59:00.837 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2/2 2023-11-16T13:59:01.099 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-16T13:59:01.398 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-16T13:59:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2/2 2023-11-16T13:59:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/2 2023-11-16T13:59:02.099 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-16T13:59:02.100 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:02.100 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:02.100 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-16T13:59:02.100 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:02.100 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:02.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2023-11-16T13:59:02.368 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-16T13:59:02.722 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-16T13:59:02.953 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-16T13:59:03.168 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 6/6 2023-11-16T13:59:03.184 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:19 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:03.465 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 6/6 2023-11-16T13:59:03.466 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 1/6 2023-11-16T13:59:03.466 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-16T13:59:03.466 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-16T13:59:03.466 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-16T13:59:03.466 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-16T13:59:04.190 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-16T13:59:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:04.191 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm-2:16.2.14-180.g9513c28f.el8.noarch 2023-11-16T13:59:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: python3-apipkg-1.5-6.el8.noarch 2023-11-16T13:59:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T13:59:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-11-16T13:59:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T13:59:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-11-16T13:59:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:04.192 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:04.311 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-fuse 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:59:04.853 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:16.2.14-180.g9513c28f.el8 ceph 131 k 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:04.854 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 131 k 2023-11-16T13:59:04.855 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 432 k 2023-11-16T13:59:04.855 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:59:05.203 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:21 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:05.325 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-16.2.14-180.g9513c28f.el8.x86_64 278 kB/s | 131 kB 00:00 2023-11-16T13:59:05.326 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:05.326 INFO:teuthology.orchestra.run.smithi112.stdout:Total 278 kB/s | 131 kB 00:00 2023-11-16T13:59:05.327 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:05.332 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:05.332 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:05.382 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:05.385 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:05.506 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:06.019 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:06.664 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:06.861 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2023-11-16T13:59:06.990 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:06.991 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-11-16T13:59:06.991 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 910 k 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:59:06.992 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:06.993 INFO:teuthology.orchestra.run.smithi061.stdout:Install 2 Packages 2023-11-16T13:59:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:06.993 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 993 k 2023-11-16T13:59:06.994 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 2.8 M 2023-11-16T13:59:06.994 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:59:07.580 INFO:teuthology.orchestra.run.smithi061.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 244 kB/s | 83 kB 00:00 2023-11-16T13:59:07.722 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:23 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:09.172 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:09.385 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:09.387 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:09.387 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:09.477 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2023-11-16T13:59:09.689 INFO:teuthology.orchestra.run.smithi061.stdout:(2/2): ceph-fuse-16.2.14-180.g9513c28f.el8.x86_ 372 kB/s | 910 kB 00:02 2023-11-16T13:59:09.689 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:09.690 INFO:teuthology.orchestra.run.smithi061.stdout:Total 368 kB/s | 993 kB 00:02 2023-11-16T13:59:09.702 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:09.726 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:09.726 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:09.823 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:09.826 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:10.002 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:10.237 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-16T13:59:10.336 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:26 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:11.510 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2/2 2023-11-16T13:59:11.936 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2/2 2023-11-16T13:59:11.936 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/2 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:59:12.017 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:16.2.14-180.g9513c28f.el8 ceph 25 k 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:12.018 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 25 k 2023-11-16T13:59:12.019 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 107 k 2023-11-16T13:59:12.019 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:59:12.171 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-16.2.14-180.g9513c28f.el8.x86_6 167 kB/s | 25 kB 00:00 2023-11-16T13:59:12.172 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:12.172 INFO:teuthology.orchestra.run.smithi112.stdout:Total 165 kB/s | 25 kB 00:00 2023-11-16T13:59:12.173 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:12.176 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:12.176 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:12.178 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:12.181 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:12.463 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:12.666 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-16T13:59:12.666 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:12.667 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:12.667 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-16T13:59:12.667 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:12.667 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:12.725 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:12.877 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados-devel 2023-11-16T13:59:13.356 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:13.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:13.357 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:13.357 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:13.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:13.357 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:13.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2023-11-16T13:59:13.769 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:29 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:14.314 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:30 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:15.565 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel x86_64 2:16.2.14-180.g9513c28f.el8 ceph 131 k 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:59:15.566 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:15.567 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 131 k 2023-11-16T13:59:15.567 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 432 k 2023-11-16T13:59:15.567 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:59:15.773 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:15.811 INFO:teuthology.orchestra.run.smithi061.stdout:librados-devel-16.2.14-180.g9513c28f.el8.x86_64 537 kB/s | 131 kB 00:00 2023-11-16T13:59:15.812 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:15.812 INFO:teuthology.orchestra.run.smithi061.stdout:Total 534 kB/s | 131 kB 00:00 2023-11-16T13:59:15.812 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:15.818 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:15.818 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:15.872 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:15.875 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:15.987 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:15.988 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:15.988 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:16.001 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:16.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2023-11-16T13:59:16.224 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:16.531 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:16.944 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:32 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:17.224 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:17.224 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:17.224 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:17.225 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:17.225 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:17.225 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:18.046 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs2 2023-11-16T13:59:18.397 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:18.612 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:18.613 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:18.613 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:18.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2023-11-16T13:59:18.952 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:34 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:19.569 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:35 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:20.502 INFO:teuthology.orchestra.run.smithi061.stdout:Package libcephfs2-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:20.733 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:20.734 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:20.734 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:20.829 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs-devel 2023-11-16T13:59:20.999 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:21.213 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:21.214 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:21.214 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:21.400 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2023-11-16T13:59:21.735 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:37 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:22.250 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:38 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:23.520 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel x86_64 2:16.2.14-180.g9513c28f.el8 ceph 25 k 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:59:23.521 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:23.522 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 25 k 2023-11-16T13:59:23.522 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 107 k 2023-11-16T13:59:23.522 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:59:23.693 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:23.758 INFO:teuthology.orchestra.run.smithi061.stdout:libcephfs-devel-16.2.14-180.g9513c28f.el8.x86_6 108 kB/s | 25 kB 00:00 2023-11-16T13:59:23.759 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:23.759 INFO:teuthology.orchestra.run.smithi061.stdout:Total 107 kB/s | 25 kB 00:00 2023-11-16T13:59:23.759 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:23.763 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:23.763 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:23.766 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:23.769 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:23.821 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:23.911 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:23.912 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:23.912 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:24.014 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:24.025 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2023-11-16T13:59:24.277 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:24.881 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:40 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:31.370 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:31.370 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:31.370 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:31.370 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:31.370 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:31.371 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:31.371 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados2 2023-11-16T13:59:31.372 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:31.372 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:31.372 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:31.372 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:31.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2023-11-16T13:59:32.257 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:48 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:32.276 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:48 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:33.705 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:33.833 INFO:teuthology.orchestra.run.smithi061.stdout:Package librados2-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:33.919 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:33.920 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-11-16T13:59:33.921 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:34.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2023-11-16T13:59:34.064 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:34.065 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:34.066 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:34.161 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librbd1 2023-11-16T13:59:34.862 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:50 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:35.050 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:51 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 88 k 2023-11-16T13:59:36.527 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:36.528 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:59:36.528 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:36.528 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:59:36.528 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:36.529 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 88 k 2023-11-16T13:59:36.529 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 240 k 2023-11-16T13:59:36.529 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:59:36.590 INFO:teuthology.orchestra.run.smithi061.stdout:Package librbd1-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:36.728 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-16.2.14-180.g9513c28f.el8.x86_64.rpm 443 kB/s | 88 kB 00:00 2023-11-16T13:59:36.728 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:36.729 INFO:teuthology.orchestra.run.smithi112.stdout:Total 440 kB/s | 88 kB 00:00 2023-11-16T13:59:36.729 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:36.738 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:36.738 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:36.786 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:36.790 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:36.821 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:36.823 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:36.823 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:36.892 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:36.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rados 2023-11-16T13:59:37.085 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:37.373 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:37.814 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:53 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:38.065 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:38.065 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:38.309 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2023-11-16T13:59:39.171 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:55 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:39.355 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rados-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:39.589 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:39.590 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:39.590 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:39.686 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rgw 2023-11-16T13:59:40.597 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:56 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:40.833 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:40.833 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:40.833 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.7 M 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.7 M 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 15 M 2023-11-16T13:59:40.834 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:59:42.151 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rgw-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:42.382 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:42.383 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:42.383 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:42.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-cephfs 2023-11-16T13:59:43.377 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:59 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:44.948 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-cephfs-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:45.096 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-16.2.14-180.g9513c28f.el8.x86_64.rpm 882 kB/s | 3.7 MB 00:04 2023-11-16T13:59:45.097 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:45.098 INFO:teuthology.orchestra.run.smithi112.stdout:Total 882 kB/s | 3.7 MB 00:04 2023-11-16T13:59:45.098 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:45.111 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:45.111 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:45.178 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:45.180 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:45.181 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:45.181 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:45.181 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:45.282 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rbd 2023-11-16T13:59:45.624 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:46.174 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:03:02 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:46.893 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:46.894 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:46.894 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:46.894 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:46.894 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:46.894 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:47.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2023-11-16T13:59:47.738 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rbd-2:16.2.14-180.g9513c28f.el8.x86_64 is already installed. 2023-11-16T13:59:47.947 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:03 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:47.969 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:47.970 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-11-16T13:59:47.971 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:48.069 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-fuse 2023-11-16T13:59:48.961 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:03:04 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 175 k 2023-11-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 175 k 2023-11-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 523 k 2023-11-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-11-16T13:59:50.124 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-16.2.14-180.g9513c28f.el8.x86_64.rpm 347 kB/s | 175 kB 00:00 2023-11-16T13:59:50.124 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:50.124 INFO:teuthology.orchestra.run.smithi112.stdout:Total 346 kB/s | 175 kB 00:00 2023-11-16T13:59:50.125 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-11-16T13:59:50.136 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-11-16T13:59:50.136 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-11-16T13:59:50.187 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-11-16T13:59:50.190 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-11-16T13:59:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-11-16T13:59:50.492 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:59:50.744 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse x86_64 2:16.2.14-180.g9513c28f.el8 ceph 88 k 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:50.745 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 88 k 2023-11-16T13:59:50.746 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 240 k 2023-11-16T13:59:50.746 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:59:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:50.930 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-fuse-16.2.14-180.g9513c28f.el8.x86_64.rpm 479 kB/s | 88 kB 00:00 2023-11-16T13:59:50.931 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:50.931 INFO:teuthology.orchestra.run.smithi061.stdout:Total 476 kB/s | 88 kB 00:00 2023-11-16T13:59:50.931 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:50.942 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:50.942 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:50.994 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:50.997 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:51.106 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:51.344 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:51.452 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:51.452 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:51.453 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-11-16T13:59:51.453 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:51.453 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T13:59:51.453 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-11-16T13:59:51.645 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:51.671 DEBUG:teuthology.parallel:result is None 2023-11-16T13:59:52.324 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:52.324 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:52.324 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:52.324 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:52.325 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:52.325 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:52.548 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-mirror 2023-11-16T13:59:53.444 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:03:09 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T13:59:55.212 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T13:59:55.212 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:55.212 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-11-16T13:59:55.213 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:55.213 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T13:59:55.213 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror x86_64 2:16.2.14-180.g9513c28f.el8 ceph 3.7 M 2023-11-16T13:59:55.214 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:55.214 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T13:59:55.214 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T13:59:55.215 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T13:59:55.215 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:55.215 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 3.7 M 2023-11-16T13:59:55.215 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 15 M 2023-11-16T13:59:55.215 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T13:59:57.440 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-mirror-16.2.14-180.g9513c28f.el8.x86_64.rpm 1.6 MB/s | 3.7 MB 00:02 2023-11-16T13:59:57.441 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T13:59:57.441 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.6 MB/s | 3.7 MB 00:02 2023-11-16T13:59:57.442 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T13:59:57.456 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T13:59:57.456 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T13:59:57.527 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T13:59:57.530 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T13:59:58.012 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T13:59:58.203 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:58.628 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T13:59:59.369 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T13:59:59.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-nbd 2023-11-16T14:00:00.473 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:03:16 ago on Thu 16 Nov 2023 01:56:44 PM UTC. 2023-11-16T14:00:02.261 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd x86_64 2:16.2.14-180.g9513c28f.el8 ceph 175 k 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-11-16T14:00:02.262 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-11-16T14:00:02.263 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-11-16T14:00:02.263 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T14:00:02.263 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 175 k 2023-11-16T14:00:02.263 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 523 k 2023-11-16T14:00:02.263 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-11-16T14:00:02.468 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-nbd-16.2.14-180.g9513c28f.el8.x86_64.rpm 855 kB/s | 175 kB 00:00 2023-11-16T14:00:02.469 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-11-16T14:00:02.469 INFO:teuthology.orchestra.run.smithi061.stdout:Total 850 kB/s | 175 kB 00:00 2023-11-16T14:00:02.469 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-11-16T14:00:02.480 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-11-16T14:00:02.480 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-11-16T14:00:02.535 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-11-16T14:00:02.538 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-11-16T14:00:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-11-16T14:00:02.993 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T14:00:03.293 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T14:00:03.980 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 1/1 2023-11-16T14:00:03.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T14:00:03.981 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-11-16T14:00:03.981 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd-2:16.2.14-180.g9513c28f.el8.x86_64 2023-11-16T14:00:03.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T14:00:03.981 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-11-16T14:00:04.203 DEBUG:teuthology.parallel:result is None 2023-11-16T14:00:04.203 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T14:00:04.203 INFO:teuthology.packaging:ref: None 2023-11-16T14:00:04.203 INFO:teuthology.packaging:tag: None 2023-11-16T14:00:04.203 INFO:teuthology.packaging:branch: pacific 2023-11-16T14:00:04.203 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T14:00:04.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-11-16T14:00:04.455 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T14:00:04.479 INFO:teuthology.orchestra.run.smithi061.stdout:16.2.14-180.g9513c28f.el8 2023-11-16T14:00:04.479 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-180.g9513c28f.el8 2023-11-16T14:00:04.479 INFO:teuthology.task.install:The correct ceph version 16.2.14-180.g9513c28f is installed. 2023-11-16T14:00:04.480 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T14:00:04.480 INFO:teuthology.packaging:ref: None 2023-11-16T14:00:04.480 INFO:teuthology.packaging:tag: None 2023-11-16T14:00:04.481 INFO:teuthology.packaging:branch: pacific 2023-11-16T14:00:04.481 INFO:teuthology.packaging:sha1: ae6ce92c0812ba06c9c623ce72df4f3fc59080b4 2023-11-16T14:00:04.481 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-11-16T14:00:04.717 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T14:00:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:16.2.14-180.g9513c28f.el8 2023-11-16T14:00:04.747 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-180.g9513c28f.el8 2023-11-16T14:00:04.747 INFO:teuthology.task.install:The correct ceph version 16.2.14-180.g9513c28f is installed. 2023-11-16T14:00:04.747 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T14:00:04.748 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:00:04.748 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T14:00:04.778 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:00:04.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T14:00:04.820 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T14:00:04.820 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:00:04.820 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T14:00:04.849 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T14:00:04.919 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:00:04.919 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T14:00:04.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T14:00:05.018 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T14:00:05.019 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:00:05.019 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T14:00:05.049 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T14:00:05.119 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:00:05.119 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T14:00:05.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T14:00:05.216 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T14:00:05.217 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:00:05.217 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T14:00:05.246 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T14:00:05.315 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:00:05.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T14:00:05.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T14:00:05.410 INFO:teuthology.run_tasks:Running task print... 2023-11-16T14:00:05.419 INFO:teuthology.task.print:**** done install task... 2023-11-16T14:00:05.420 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T14:00:05.507 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ae6ce92c0812ba06c9c623ce72df4f3fc59080b4'} 2023-11-16T14:00:05.507 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2023-11-16T14:00:05.508 INFO:tasks.cephadm:Cluster fsid is 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:05.508 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T14:00:05.508 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-16T14:00:05.508 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi061': '172.21.15.61', 'mon.smithi112': '172.21.15.112'} 2023-11-16T14:00:05.508 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T14:00:05.508 DEBUG:teuthology.orchestra.run.smithi061:> sudo hostname $(hostname -s) 2023-11-16T14:00:05.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2023-11-16T14:00:05.566 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2023-11-16T14:00:05.567 DEBUG:teuthology.orchestra.run.smithi061:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:05.988 INFO:teuthology.orchestra.run.smithi061.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Nov 16 14:00 /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:05.989 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:06.333 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Nov 16 14:00 /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:06.333 DEBUG:teuthology.orchestra.run.smithi061:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:06.357 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T14:00:06.388 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2023-11-16T14:00:06.389 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2023-11-16T14:00:06.400 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2023-11-16T14:00:07.422 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-11-16T14:00:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-11-16T14:00:33.366 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-11-16T14:00:33.367 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-11-16T14:00:36.626 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2023-11-16T14:00:36.626 INFO:teuthology.orchestra.run.smithi061.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2023-11-16T14:00:36.626 INFO:teuthology.orchestra.run.smithi061.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2023-11-16T14:00:36.626 INFO:teuthology.orchestra.run.smithi061.stdout: "repo_digests": [ 2023-11-16T14:00:36.627 INFO:teuthology.orchestra.run.smithi061.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2023-11-16T14:00:36.627 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2023-11-16T14:00:36.627 INFO:teuthology.orchestra.run.smithi061.stdout:} 2023-11-16T14:00:36.651 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph 2023-11-16T14:00:36.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2023-11-16T14:00:36.720 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /etc/ceph 2023-11-16T14:00:36.750 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2023-11-16T14:00:36.791 INFO:tasks.cephadm:Writing seed config... 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T14:00:36.793 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T14:00:36.794 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-16T14:00:36.795 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T14:00:36.796 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T14:00:36.797 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T14:00:36.797 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T14:00:36.797 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T14:00:36.797 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:00:36.797 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T14:00:36.816 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 71d22a3a-8488-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T14:00:36.816 DEBUG:teuthology.orchestra.run.smithi061:mon.smithi061> sudo journalctl -f -n 0 -u ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service 2023-11-16T14:00:36.858 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T14:00:36.859 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.61 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T14:00:36.886 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:-- Logs begin at Thu 2023-11-16 13:47:48 UTC. -- 2023-11-16T14:00:37.070 INFO:teuthology.orchestra.run.smithi061.stderr:-------------------------------------------------------------------------------- 2023-11-16T14:00:37.071 INFO:teuthology.orchestra.run.smithi061.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '71d22a3a-8488-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.61', '--skip-admin-label'] 2023-11-16T14:00:37.092 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-11-16T14:00:37.107 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying podman|docker is present... 2023-11-16T14:00:37.127 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-11-16T14:00:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying lvm2 is present... 2023-11-16T14:00:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying time synchronization is in place... 2023-11-16T14:00:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T14:00:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:00:37.154 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: enabled 2023-11-16T14:00:37.160 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: active 2023-11-16T14:00:37.160 INFO:teuthology.orchestra.run.smithi061.stderr:Unit chronyd.service is enabled and running 2023-11-16T14:00:37.160 INFO:teuthology.orchestra.run.smithi061.stderr:Repeating the final host check... 2023-11-16T14:00:37.180 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-11-16T14:00:37.194 INFO:teuthology.orchestra.run.smithi061.stderr:podman (/bin/podman) version 4.6.1 is present 2023-11-16T14:00:37.194 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl is present 2023-11-16T14:00:37.194 INFO:teuthology.orchestra.run.smithi061.stderr:lvcreate is present 2023-11-16T14:00:37.199 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T14:00:37.205 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:00:37.211 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: enabled 2023-11-16T14:00:37.217 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: active 2023-11-16T14:00:37.218 INFO:teuthology.orchestra.run.smithi061.stderr:Unit chronyd.service is enabled and running 2023-11-16T14:00:37.218 INFO:teuthology.orchestra.run.smithi061.stderr:Host looks OK 2023-11-16T14:00:37.218 INFO:teuthology.orchestra.run.smithi061.stderr:Cluster fsid: 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:37.218 INFO:teuthology.orchestra.run.smithi061.stderr:Acquiring lock 140189608884712 on /run/cephadm/71d22a3a-8488-11ee-8dbe-212e2dc638e7.lock 2023-11-16T14:00:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:Lock 140189608884712 acquired on /run/cephadm/71d22a3a-8488-11ee-8dbe-212e2dc638e7.lock 2023-11-16T14:00:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying IP 172.21.15.61 port 3300 ... 2023-11-16T14:00:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying IP 172.21.15.61 port 6789 ... 2023-11-16T14:00:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:Base mon IP(s) is [172.21.15.61:3300, 172.21.15.61:6789], mon addrv is [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-11-16T14:00:37.222 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.61 metric 100 2023-11-16T14:00:37.222 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.61 metric 100 2023-11-16T14:00:37.225 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T14:00:37.225 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-16T14:00:37.225 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: inet6 fe80::ec4:7aff:feb6:f8b9/64 scope link 2023-11-16T14:00:37.228 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-16T14:00:37.229 INFO:teuthology.orchestra.run.smithi061.stderr:Mon IP `172.21.15.61` is in CIDR network `172.21.0.0/20` 2023-11-16T14:00:37.229 INFO:teuthology.orchestra.run.smithi061.stderr:Mon IP `172.21.15.61` is in CIDR network `172.21.0.0/20` 2023-11-16T14:00:37.229 INFO:teuthology.orchestra.run.smithi061.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-16T14:00:37.230 INFO:teuthology.orchestra.run.smithi061.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T14:00:37.231 INFO:teuthology.orchestra.run.smithi061.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-11-16T14:00:37.310 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2023-11-16T14:00:43.815 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Getting image source signatures 2023-11-16T14:00:43.815 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2023-11-16T14:00:43.815 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-16T14:00:43.815 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2023-11-16T14:00:43.876 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Writing manifest to image destination 2023-11-16T14:00:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2023-11-16T14:00:44.359 INFO:teuthology.orchestra.run.smithi061.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2023-11-16T14:00:44.707 INFO:teuthology.orchestra.run.smithi061.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2023-11-16T14:00:44.707 INFO:teuthology.orchestra.run.smithi061.stderr:Extracting ceph user uid/gid from container image... 2023-11-16T14:00:44.965 INFO:teuthology.orchestra.run.smithi061.stderr:stat: 167 167 2023-11-16T14:00:45.299 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial keys... 2023-11-16T14:00:45.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCNIFZlvUXbHxAAxmBOI7dl3cPt7l1R2VVUrg== 2023-11-16T14:00:46.124 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCOIFZlED/+BhAA3Bou8YAjTWafCaCzJxjnAw== 2023-11-16T14:00:46.716 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCOIFZl2kALKhAAixF2NY/a9veuSHIrktPEPw== 2023-11-16T14:00:47.041 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial monmap... 2023-11-16T14:00:47.307 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T14:00:47.308 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:47.308 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T14:00:47.632 INFO:teuthology.orchestra.run.smithi061.stderr:monmaptool for smithi061 [v2:172.21.15.61:3300,v1:172.21.15.61:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T14:00:47.632 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: set fsid to 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:47.633 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T14:00:47.633 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:00:47.633 INFO:teuthology.orchestra.run.smithi061.stderr:Creating mon... 2023-11-16T14:00:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.891+0000 7f980a1f5700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-16T14:00:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 1 imported monmap: 2023-11-16T14:00:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-16T14:00:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: last_changed 2023-11-16T14:00:47.298126+0000 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: created 2023-11-16T14:00:47.298126+0000 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.smithi061 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 0 /usr/bin/ceph-mon: set fsid to 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: RocksDB version: 6.8.1 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Compile date Jul 8 2021 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: DB SUMMARY 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi061/store.db dir, Total Num: 0, files: 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi061/store.db: 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.error_if_exists: 0 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.create_if_missing: 1 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.paranoid_checks: 1 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.env: 0x563d3d5521c0 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.fs: Posix File System 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.info_log: 0x563d3e767000 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.statistics: (nil) 2023-11-16T14:00:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.use_fsync: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_log_file_size: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.allow_fallocate: 1 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.use_direct_reads: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.db_log_dir: 2023-11-16T14:00:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi061/store.db 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_subcompactions: 1 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_background_flushes: -1 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.write_buffer_manager: 0x563d3e76f530 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T14:00:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.unordered_write: 0 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T14:00:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.row_cache: None 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.wal_filter: None 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.preserve_deletes: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.two_write_queues: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.atomic_flush: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.log_readahead_size: 0 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-16T14:00:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_background_jobs: 2 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_background_compactions: -1 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.max_open_files: -1 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T14:00:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Compression algorithms supported: 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kZSTD supported: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kXpressCompression supported: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kLZ4Compression supported: 1 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kBZip2Compression supported: 0 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kZlibCompression supported: 1 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: kSnappyCompression supported: 1 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.892+0000 7f980a1f5700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000001 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.merge_operator: 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_filter: None 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d3e680160) 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T14:00:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache: 0x563d3e6b6d10 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-16T14:00:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-16T14:00:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression: NoCompression 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.num_levels: 7 2023-11-16T14:00:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T14:00:47.936 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T14:00:47.936 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T14:00:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T14:00:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T14:00:47.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.table_properties_collectors: 2023-11-16T14:00:47.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.inplace_update_support: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.bloom_locality: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.max_successive_merges: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.ttl: 2592000 2023-11-16T14:00:47.941 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.893+0000 7f980a1f5700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.894+0000 7f980a1f5700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.894+0000 7f980a1f5700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.894+0000 7f980a1f5700 4 rocksdb: DB pointer 0x563d3e77b800 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.894+0000 7f97f307e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.894+0000 7f97f307e700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T14:00:47.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T14:00:47.943 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-16T14:00:47.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:47.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.896+0000 7f980a1f5700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.896+0000 7f980a1f5700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-16T14:00:47.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-11-16T14:00:47.896+0000 7f980a1f5700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi061 for mon.smithi061 2023-11-16T14:00:48.228 INFO:teuthology.orchestra.run.smithi061.stderr:create mon.smithi061 on 2023-11-16T14:00:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T14:00:48.340 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T14:00:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target. 2023-11-16T14:00:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target. 2023-11-16T14:00:48.655 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to reset failed state of unit ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service: Unit ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service not loaded. 2023-11-16T14:00:48.662 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target.wants/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service → /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T14:00:49.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:48 smithi061 systemd[1]: Starting Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7... 2023-11-16T14:00:49.145 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-11-16T14:00:49.153 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:00:49.154 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-11-16T14:00:49.154 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to enable service . firewalld.service is not available 2023-11-16T14:00:49.154 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mon to start... 2023-11-16T14:00:49.154 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mon... 2023-11-16T14:00:49.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:49 smithi061 bash[111398]: b2e1efc98acb074ae70ee9a4d81e1be7fffc5bd2559f2534bbf039fd9b15311a 2023-11-16T14:00:49.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:49 smithi061 systemd[1]: Started Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7. 2023-11-16T14:00:49.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.167586+0000 mon.smithi061 (mon.0) 0 : [INF] mkfs 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:49.509 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.144038+0000 mon.smithi061 (mon.0) 1 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-11-16T14:00:49.537 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.533+0000 7f296945b700 1 Processor -- start 2023-11-16T14:00:49.537 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.533+0000 7f296945b700 1 -- start start 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f296945b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f296945b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29640fc7a0 con 0x7f29640fbe90 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f2962ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f2962ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46614/0 (socket says 172.21.15.61:46614) 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f2962ffd700 1 -- 172.21.15.61:0/359903625 learned_addr learned my addr 172.21.15.61:0/359903625 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.534+0000 7f2962ffd700 1 -- 172.21.15.61:0/359903625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29640fc8e0 con 0x7f29640fbe90 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f2962ffd700 1 --2- 172.21.15.61:0/359903625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2954009960 tx=0x7f29540092d0).ready entity=mon.0 client_cookie=224a1ed5d73595b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f2961ffb700 1 -- 172.21.15.61:0/359903625 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295400b9c0 con 0x7f29640fbe90 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f2961ffb700 1 -- 172.21.15.61:0/359903625 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295400bb20 con 0x7f29640fbe90 2023-11-16T14:00:49.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f2961ffb700 1 -- 172.21.15.61:0/359903625 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295400bdf0 con 0x7f29640fbe90 2023-11-16T14:00:49.540 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 -- 172.21.15.61:0/359903625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 msgr2=0x7f29640fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:49.540 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 --2- 172.21.15.61:0/359903625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2954009960 tx=0x7f29540092d0).stop 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 -- 172.21.15.61:0/359903625 shutdown_connections 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 --2- 172.21.15.61:0/359903625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f29640fc2b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 -- 172.21.15.61:0/359903625 >> 172.21.15.61:0/359903625 conn(0x7f29640f7630 msgr2=0x7f29640f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 -- 172.21.15.61:0/359903625 shutdown_connections 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.535+0000 7f296945b700 1 -- 172.21.15.61:0/359903625 wait complete. 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.536+0000 7f296945b700 1 Processor -- start 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.536+0000 7f296945b700 1 -- start start 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f296945b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f296945b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296410a940 con 0x7f29640fbe90 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f2962ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:49.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f2962ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46626/0 (socket says 172.21.15.61:46626) 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f2962ffd700 1 -- 172.21.15.61:0/1512722338 learned_addr learned my addr 172.21.15.61:0/1512722338 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f2962ffd700 1 -- 172.21.15.61:0/1512722338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2954009570 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f2962ffd700 1 --2- 172.21.15.61:0/1512722338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2954009130 tx=0x7f2954017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295401c420 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295401c580 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296410ab40 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295401c850 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.537+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296410afd0 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.538+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f295402c070 con 0x7f29640fbe90 2023-11-16T14:00:49.542 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.538+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2954025a50 con 0x7f29640fbe90 2023-11-16T14:00:49.543 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.538+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2964045bb0 con 0x7f29640fbe90 2023-11-16T14:00:49.543 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.541+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f29540449c0 con 0x7f29640fbe90 2023-11-16T14:00:49.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.611+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2964103ed0 con 0x7f29640fbe90 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.611+0000 7f295b7fe700 1 -- 172.21.15.61:0/1512722338 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f2954033030 con 0x7f29640fbe90 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: cluster: 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: id: 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: services: 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi061 (age 0.451639s) 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: data: 2023-11-16T14:00:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: pgs: 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 msgr2=0x7f296410a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 --2- 172.21.15.61:0/1512722338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2954009130 tx=0x7f2954017000).stop 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 shutdown_connections 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 --2- 172.21.15.61:0/1512722338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29640fbe90 0x7f296410a450 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 >> 172.21.15.61:0/1512722338 conn(0x7f29640f7630 msgr2=0x7f29640fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 shutdown_connections 2023-11-16T14:00:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:49.612+0000 7f296945b700 1 -- 172.21.15.61:0/1512722338 wait complete. 2023-11-16T14:00:49.959 INFO:teuthology.orchestra.run.smithi061.stderr:mon is available 2023-11-16T14:00:49.959 INFO:teuthology.orchestra.run.smithi061.stderr:Assimilating anything we can from ceph.conf... 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.160634+0000 mon.smithi061 (mon.0) 2 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.160864+0000 mon.smithi061 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi061=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.169953+0000 mon.smithi061 (mon.0) 4 : cluster [DBG] fsmap 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.172175+0000 mon.smithi061 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: cluster 2023-11-16T14:00:49.172751+0000 mon.smithi061 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-16T14:00:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: audit 2023-11-16T14:00:49.612243+0000 mon.smithi061 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.61:0/1512722338' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T14:00:50.335 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.332+0000 7f8fb6dc9700 1 Processor -- start 2023-11-16T14:00:50.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.332+0000 7f8fb6dc9700 1 -- start start 2023-11-16T14:00:50.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb6dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb6dc9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb00ff000 con 0x7f8fb0027670 2023-11-16T14:00:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb4b65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb4b65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46628/0 (socket says 172.21.15.61:46628) 2023-11-16T14:00:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb4b65700 1 -- 172.21.15.61:0/1637321736 learned_addr learned my addr 172.21.15.61:0/1637321736 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb4b65700 1 -- 172.21.15.61:0/1637321736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb00ff140 con 0x7f8fb0027670 2023-11-16T14:00:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8fb4b65700 1 --2- 172.21.15.61:0/1637321736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8fa400b5f0 tx=0x7f8fa4006ee0).ready entity=mon.0 client_cookie=6dfd18674ea7bd7c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8faf7fe700 1 -- 172.21.15.61:0/1637321736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fa400d600 con 0x7f8fb0027670 2023-11-16T14:00:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8faf7fe700 1 -- 172.21.15.61:0/1637321736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa400d760 con 0x7f8fb0027670 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.333+0000 7f8faf7fe700 1 -- 172.21.15.61:0/1637321736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fa400da30 con 0x7f8fb0027670 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1637321736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 msgr2=0x7f8fb00feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 --2- 172.21.15.61:0/1637321736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8fa400b5f0 tx=0x7f8fa4006ee0).stop 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1637321736 shutdown_connections 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 --2- 172.21.15.61:0/1637321736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00feb10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1637321736 >> 172.21.15.61:0/1637321736 conn(0x7f8fb00f6590 msgr2=0x7f8fb00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1637321736 shutdown_connections 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.334+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1637321736 wait complete. 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb6dc9700 1 Processor -- start 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb6dc9700 1 -- start start 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb6dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb6dc9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb00faa10 con 0x7f8fb0027670 2023-11-16T14:00:50.340 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb4b65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:50.340 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb4b65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46640/0 (socket says 172.21.15.61:46640) 2023-11-16T14:00:50.340 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.335+0000 7f8fb4b65700 1 -- 172.21.15.61:0/1256011076 learned_addr learned my addr 172.21.15.61:0/1256011076 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fb4b65700 1 -- 172.21.15.61:0/1256011076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa400b200 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fb4b65700 1 --2- 172.21.15.61:0/1256011076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8fa4006c70 tx=0x7f8fa4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fa4011b40 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa4011ca0 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb00fabb0 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fa401d430 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb00fb040 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f8fa400f070 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.336+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8fa401dbf0 con 0x7f8fb0027670 2023-11-16T14:00:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.337+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb0045bb0 con 0x7f8fb0027670 2023-11-16T14:00:50.342 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.340+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8fa4016070 con 0x7f8fb0027670 2023-11-16T14:00:50.409 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.407+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8fb00ff3f0 con 0x7f8fb0027670 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.410+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8fa401de80 con 0x7f8fb0027670 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.410+0000 7f8fad7fa700 1 -- 172.21.15.61:0/1256011076 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f8fa40361e0 con 0x7f8fb0027670 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [global] 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: fsid = 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: log_to_journald = false 2023-11-16T14:00:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-11-16T14:00:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-16T14:00:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-16T14:00:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-16T14:00:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [mgr] 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [osd] 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-16T14:00:50.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 msgr2=0x7f8fb00fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 --2- 172.21.15.61:0/1256011076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8fa4006c70 tx=0x7f8fa4016000).stop 2023-11-16T14:00:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 shutdown_connections 2023-11-16T14:00:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 --2- 172.21.15.61:0/1256011076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fb0027670 0x7f8fb00fc590 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 >> 172.21.15.61:0/1256011076 conn(0x7f8fb00f6590 msgr2=0x7f8fb00f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 shutdown_connections 2023-11-16T14:00:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:50.412+0000 7f8fb6dc9700 1 -- 172.21.15.61:0/1256011076 wait complete. 2023-11-16T14:00:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:Generating new minimal ceph.conf... 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8afc9cf700 1 Processor -- start 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8afc9cf700 1 -- start start 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8afc9cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8afc9cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af80fde90 con 0x7f8af80fd580 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8af659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8af659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46648/0 (socket says 172.21.15.61:46648) 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.102+0000 7f8af659c700 1 -- 172.21.15.61:0/2941537381 learned_addr learned my addr 172.21.15.61:0/2941537381 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.103+0000 7f8af659c700 1 -- 172.21.15.61:0/2941537381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af80fdfd0 con 0x7f8af80fd580 2023-11-16T14:00:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.103+0000 7f8af659c700 1 --2- 172.21.15.61:0/2941537381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8ae800b5f0 tx=0x7f8ae8006ee0).ready entity=mon.0 client_cookie=6c0e5c761578d331 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.103+0000 7f8af559a700 1 -- 172.21.15.61:0/2941537381 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae800d600 con 0x7f8af80fd580 2023-11-16T14:00:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.103+0000 7f8af559a700 1 -- 172.21.15.61:0/2941537381 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8ae800dbc0 con 0x7f8af80fd580 2023-11-16T14:00:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.103+0000 7f8af559a700 1 -- 172.21.15.61:0/2941537381 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae8011b90 con 0x7f8af80fd580 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- 172.21.15.61:0/2941537381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 msgr2=0x7f8af80fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 --2- 172.21.15.61:0/2941537381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8ae800b5f0 tx=0x7f8ae8006ee0).stop 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- 172.21.15.61:0/2941537381 shutdown_connections 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 --2- 172.21.15.61:0/2941537381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af80fd9a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- 172.21.15.61:0/2941537381 >> 172.21.15.61:0/2941537381 conn(0x7f8af80f5ce0 msgr2=0x7f8af80f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- 172.21.15.61:0/2941537381 shutdown_connections 2023-11-16T14:00:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- 172.21.15.61:0/2941537381 wait complete. 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 Processor -- start 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.104+0000 7f8afc9cf700 1 -- start start 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8afc9cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8afc9cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af810b220 con 0x7f8af80fd580 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8af659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8af659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46654/0 (socket says 172.21.15.61:46654) 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8af659c700 1 -- 172.21.15.61:0/4167760467 learned_addr learned my addr 172.21.15.61:0/4167760467 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8af659c700 1 -- 172.21.15.61:0/4167760467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae800b200 con 0x7f8af80fd580 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8af659c700 1 --2- 172.21.15.61:0/4167760467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8ae8006800 tx=0x7f8ae8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae800de50 con 0x7f8af80fd580 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af810b420 con 0x7f8af80fd580 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8ae801b870 con 0x7f8af80fd580 2023-11-16T14:00:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.105+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae801a7a0 con 0x7f8af80fd580 2023-11-16T14:00:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.106+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af810bfd0 con 0x7f8af80fd580 2023-11-16T14:00:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.106+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f8ae800f070 con 0x7f8af80fd580 2023-11-16T14:00:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.106+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8ae8024c40 con 0x7f8af80fd580 2023-11-16T14:00:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.107+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af80591e0 con 0x7f8af80fd580 2023-11-16T14:00:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.109+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8ae801f030 con 0x7f8af80fd580 2023-11-16T14:00:51.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.175+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8af80faa10 con 0x7f8af80fd580 2023-11-16T14:00:51.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.175+0000 7f8aeeffd700 1 -- 172.21.15.61:0/4167760467 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f8ae8033090 con 0x7f8af80fd580 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.176+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 msgr2=0x7f8af810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.176+0000 7f8afc9cf700 1 --2- 172.21.15.61:0/4167760467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8ae8006800 tx=0x7f8ae8006ee0).stop 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.176+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 shutdown_connections 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.176+0000 7f8afc9cf700 1 --2- 172.21.15.61:0/4167760467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80fd580 0x7f8af810ad30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.176+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 >> 172.21.15.61:0/4167760467 conn(0x7f8af80f5ce0 msgr2=0x7f8af80ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.177+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 shutdown_connections 2023-11-16T14:00:51.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:51.177+0000 7f8afc9cf700 1 -- 172.21.15.61:0/4167760467 wait complete. 2023-11-16T14:00:51.444 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: audit 2023-11-16T14:00:50.408421+0000 mon.smithi061 (mon.0) 8 : audit [INF] from='client.? 172.21.15.61:0/1256011076' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T14:00:51.444 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: audit 2023-11-16T14:00:50.411932+0000 mon.smithi061 (mon.0) 9 : audit [INF] from='client.? 172.21.15.61:0/1256011076' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T14:00:51.518 INFO:teuthology.orchestra.run.smithi061.stderr:Restarting the monitor... 2023-11-16T14:00:51.704 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 systemd[1]: Stopping Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7... 2023-11-16T14:00:51.704 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 bash[111737]: Error: no container with name or ID "ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon.smithi061" found: no such container 2023-11-16T14:00:51.704 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: debug 2023-11-16T14:00:51.701+0000 7f3ef7d38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi061 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-16T14:00:52.003 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111413]: debug 2023-11-16T14:00:51.701+0000 7f3ef7d38700 -1 mon.smithi061@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T14:00:52.003 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:51 smithi061 bash[111776]: ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061 2023-11-16T14:00:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 systemd[1]: ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service: Succeeded. 2023-11-16T14:00:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 systemd[1]: Stopped Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7. 2023-11-16T14:00:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 systemd[1]: Starting Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7... 2023-11-16T14:00:52.494 INFO:teuthology.orchestra.run.smithi061.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.487+0000 7f188b150700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.487+0000 7f188b150700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 bash[111896]: e569e09c2e7ce8fcf32195ae3abe5e6508332a4c811728bec2cb07e67741644f 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.487+0000 7f188b150700 0 pidfile_write: ignore empty --pid-file 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 systemd[1]: Started Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7. 2023-11-16T14:00:52.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.492+0000 7f188b150700 0 load: jerasure load: lrc load: isa 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: RocksDB version: 6.8.1 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Compile date Jul 8 2021 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: DB SUMMARY 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: CURRENT file: CURRENT 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-16T14:00:52.760 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi061/store.db dir, Total Num: 1, files: 000004.sst 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi061/store.db: 000006.log size: 94238 ; 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.error_if_exists: 0 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.create_if_missing: 0 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.paranoid_checks: 1 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.env: 0x55b4f2a5f1c0 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.fs: Posix File System 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.info_log: 0x55b4f3c39a40 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-16T14:00:52.761 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.statistics: (nil) 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.use_fsync: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_log_file_size: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.allow_fallocate: 1 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-16T14:00:52.762 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.use_direct_reads: 0 2023-11-16T14:00:52.763 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T14:00:52.763 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-16T14:00:52.763 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.db_log_dir: 2023-11-16T14:00:52.763 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi061/store.db 2023-11-16T14:00:52.763 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_subcompactions: 1 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_background_flushes: -1 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T14:00:52.765 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.write_buffer_manager: 0x55b4f48b0030 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-16T14:00:52.766 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.unordered_write: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.row_cache: None 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.wal_filter: None 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.preserve_deletes: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.two_write_queues: 0 2023-11-16T14:00:52.767 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.atomic_flush: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.log_readahead_size: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_background_jobs: 2 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_background_compactions: -1 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T14:00:52.768 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_open_files: -1 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Compression algorithms supported: 2023-11-16T14:00:52.769 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kZSTD supported: 0 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kXpressCompression supported: 0 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kLZ4Compression supported: 1 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kBZip2Compression supported: 0 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kZlibCompression supported: 1 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: kSnappyCompression supported: 1 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000005 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-16T14:00:52.770 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.merge_operator: 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_filter: None 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b4f3b53cc0) 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cache_index_and_filter_blocks: 1 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: pin_top_level_index_and_filter: 1 2023-11-16T14:00:52.771 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: index_type: 0 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: data_block_index_type: 0 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: index_shortening: 1 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: data_block_hash_table_util_ratio: 0.750000 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: hash_index_allow_collision: 1 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: checksum: 1 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: no_block_cache: 0 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_cache: 0x55b4f3b88f10 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_cache_name: BinnedLRUCache 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_cache_options: 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: capacity : 536870912 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: num_shard_bits : 4 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: strict_capacity_limit : 0 2023-11-16T14:00:52.772 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: high_pri_pool_ratio: 0.000 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_cache_compressed: (nil) 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: persistent_cache: (nil) 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_size: 4096 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_size_deviation: 10 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_restart_interval: 16 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: index_block_restart_interval: 1 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: metadata_block_size: 4096 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: partition_filters: 0 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: use_delta_encoding: 1 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: whole_key_filtering: 1 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: verify_compression: 0 2023-11-16T14:00:52.773 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: read_amp_bytes_per_bit: 0 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: format_version: 2 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: enable_index_compression: 1 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: block_align: 0 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression: NoCompression 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.num_levels: 7 2023-11-16T14:00:52.774 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T14:00:52.775 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T14:00:52.776 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T14:00:52.777 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T14:00:52.778 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.table_properties_collectors: 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.inplace_update_support: 0 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T14:00:52.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.bloom_locality: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.max_successive_merges: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.ttl: 2592000 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.493+0000 7f188b150700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.494+0000 7f188b150700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.494+0000 7f188b150700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.494+0000 7f188b150700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700143252495358, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.494+0000 7f188b150700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.494+0000 7f188b150700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.495+0000 7f188b150700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700143252496288, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 89832, "table_properties": {"data_size": 87947, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12755, "raw_average_key_size": 50, "raw_value_size": 80933, "raw_average_value_size": 317, "num_data_blocks": 11, "num_entries": 255, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700143252, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.495+0000 7f188b150700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.495+0000 7f188b150700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700143252496707, "job": 1, "event": "recovery_finished"} 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.496+0000 7f188b150700 4 rocksdb: DB pointer 0x55b4f3c4d800 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.496+0000 7f1871f58700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.496+0000 7f1871f58700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** DB Stats ** 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:52.781 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** Compaction Stats [default] ** 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: L0 2/0 89.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.782 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Sum 2/0 89.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** Compaction Stats [default] ** 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(Keys): cumulative 0, interval 0 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Cumulative compaction: 0.00 GB write, 28.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:52.783 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Interval compaction: 0.00 GB write, 28.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** File Read Latency Histogram By Level [default] ** 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** Compaction Stats [default] ** 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: L0 2/0 89.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Sum 2/0 89.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** Compaction Stats [default] ** 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T14:00:52.784 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.0 0.00 0.00 1 0.001 0 0 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: AddFile(Keys): cumulative 0, interval 0 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Cumulative compaction: 0.00 GB write, 28.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T14:00:52.785 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ** File Read Latency Histogram By Level [default] ** 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.497+0000 7f188b150700 0 starting mon.smithi061 rank 0 at public addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] at bind addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi061 fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 1 mon.smithi061@-1(???) e1 preinit fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).mds e1 new map 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).mds e1 print_map 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: e1 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: legacy client fscid: -1 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.786 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: No filesystems configured 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:50.408421+0000 mon.smithi061 (mon.0) 8 : audit [INF] from='client.? 172.21.15.61:0/1256011076' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:50.411932+0000 mon.smithi061 (mon.0) 9 : audit [INF] from='client.? 172.21.15.61:0/1256011076' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:00:52.498+0000 7f188b150700 1 mon.smithi061@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:52.514717+0000 mon.smithi061 (mon.0) 1 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:52.514814+0000 mon.smithi061 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi061=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:52.514870+0000 mon.smithi061 (mon.0) 3 : cluster [DBG] fsmap 2023-11-16T14:00:52.787 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:52.514910+0000 mon.smithi061 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-16T14:00:52.788 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:52.515405+0000 mon.smithi061 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-16T14:00:52.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.825+0000 7fab4e17e700 1 Processor -- start 2023-11-16T14:00:52.832 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.825+0000 7fab4e17e700 1 -- start start 2023-11-16T14:00:52.832 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.825+0000 7fab4e17e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:52.832 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.825+0000 7fab4e17e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480fb580 con 0x7fab480fac70 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab4d17c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab4d17c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46668/0 (socket says 172.21.15.61:46668) 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab4d17c700 1 -- 172.21.15.61:0/2934629461 learned_addr learned my addr 172.21.15.61:0/2934629461 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab4d17c700 1 -- 172.21.15.61:0/2934629461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480fb6c0 con 0x7fab480fac70 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab4d17c700 1 --2- 172.21.15.61:0/2934629461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fab4400b5f0 tx=0x7fab44006ee0).ready entity=mon.0 client_cookie=d46e12d6bb78e18a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab37fff700 1 -- 172.21.15.61:0/2934629461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab4400d600 con 0x7fab480fac70 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab37fff700 1 -- 172.21.15.61:0/2934629461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fab4400dbc0 con 0x7fab480fac70 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.826+0000 7fab37fff700 1 -- 172.21.15.61:0/2934629461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab44011b90 con 0x7fab480fac70 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 -- 172.21.15.61:0/2934629461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 msgr2=0x7fab480fb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 --2- 172.21.15.61:0/2934629461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fab4400b5f0 tx=0x7fab44006ee0).stop 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 -- 172.21.15.61:0/2934629461 shutdown_connections 2023-11-16T14:00:52.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 --2- 172.21.15.61:0/2934629461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab480fb090 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:52.835 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 -- 172.21.15.61:0/2934629461 >> 172.21.15.61:0/2934629461 conn(0x7fab480f63c0 msgr2=0x7fab480f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:52.835 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 -- 172.21.15.61:0/2934629461 shutdown_connections 2023-11-16T14:00:52.835 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 -- 172.21.15.61:0/2934629461 wait complete. 2023-11-16T14:00:52.835 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.827+0000 7fab4e17e700 1 Processor -- start 2023-11-16T14:00:52.835 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4e17e700 1 -- start start 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4e17e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4e17e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4810afc0 con 0x7fab480fac70 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4d17c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4d17c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46684/0 (socket says 172.21.15.61:46684) 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4d17c700 1 -- 172.21.15.61:0/766864520 learned_addr learned my addr 172.21.15.61:0/766864520 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4d17c700 1 -- 172.21.15.61:0/766864520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4400b200 con 0x7fab480fac70 2023-11-16T14:00:52.836 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab4d17c700 1 --2- 172.21.15.61:0/766864520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fab44000c00 tx=0x7fab44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab4401b410 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.828+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fab4401b9d0 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.829+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4810b1c0 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.829+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab4401a620 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.829+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4810b650 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.829+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fab4402b070 con 0x7fab480fac70 2023-11-16T14:00:52.837 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.829+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fab44024a90 con 0x7fab480fac70 2023-11-16T14:00:52.838 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.830+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48045bb0 con 0x7fab480fac70 2023-11-16T14:00:52.838 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.832+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fab4401f020 con 0x7fab480fac70 2023-11-16T14:00:52.901 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.898+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fab480fc9b0 con 0x7fab480fac70 2023-11-16T14:00:52.904 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.901+0000 7fab35ffb700 1 -- 172.21.15.61:0/766864520 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fab44016070 con 0x7fab480fac70 2023-11-16T14:00:52.905 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 msgr2=0x7fab4810aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:52.905 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 --2- 172.21.15.61:0/766864520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fab44000c00 tx=0x7fab44006ee0).stop 2023-11-16T14:00:52.905 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 shutdown_connections 2023-11-16T14:00:52.905 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 --2- 172.21.15.61:0/766864520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab480fac70 0x7fab4810aad0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:52.905 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 >> 172.21.15.61:0/766864520 conn(0x7fab480f63c0 msgr2=0x7fab4806a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:52.906 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 shutdown_connections 2023-11-16T14:00:52.906 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:52.902+0000 7fab4e17e700 1 -- 172.21.15.61:0/766864520 wait complete. 2023-11-16T14:00:53.234 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-16T14:00:53.234 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T14:00:53.235 INFO:teuthology.orchestra.run.smithi061.stderr:Creating mgr... 2023-11-16T14:00:53.235 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying port 9283 ... 2023-11-16T14:00:53.368 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to reset failed state of unit ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mgr.smithi061.zcjido.service: Unit ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mgr.smithi061.zcjido.service not loaded. 2023-11-16T14:00:53.374 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7.target.wants/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mgr.smithi061.zcjido.service → /etc/systemd/system/ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T14:00:53.888 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-11-16T14:00:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:00:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-11-16T14:00:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to enable service . firewalld.service is not available 2023-11-16T14:00:53.904 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-11-16T14:00:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:00:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-11-16T14:00:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-16T14:00:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr to start... 2023-11-16T14:00:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr... 2023-11-16T14:00:54.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:52.902867+0000 mon.smithi061 (mon.0) 6 : audit [INF] from='client.? 172.21.15.61:0/766864520' entity='client.admin' 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3ce10ba700 1 Processor -- start 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3ce10ba700 1 -- start start 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3ce10ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3ce10ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc0fc960 con 0x7f3cdc027610 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3cdad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3cdad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46720/0 (socket says 172.21.15.61:46720) 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3cdad9d700 1 -- 172.21.15.61:0/1985619245 learned_addr learned my addr 172.21.15.61:0/1985619245 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.460+0000 7f3cdad9d700 1 -- 172.21.15.61:0/1985619245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc101460 con 0x7f3cdc027610 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3cdad9d700 1 --2- 172.21.15.61:0/1985619245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3ccc00b5f0 tx=0x7f3ccc006ee0).ready entity=mon.0 client_cookie=42caec5aa59becc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3cd9d9b700 1 -- 172.21.15.61:0/1985619245 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc00d600 con 0x7f3cdc027610 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3cd9d9b700 1 -- 172.21.15.61:0/1985619245 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3ccc00dbc0 con 0x7f3cdc027610 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3cd9d9b700 1 -- 172.21.15.61:0/1985619245 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc011b90 con 0x7f3cdc027610 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 -- 172.21.15.61:0/1985619245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 msgr2=0x7f3cdc100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 --2- 172.21.15.61:0/1985619245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3ccc00b5f0 tx=0x7f3ccc006ee0).stop 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 -- 172.21.15.61:0/1985619245 shutdown_connections 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 --2- 172.21.15.61:0/1985619245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc100f70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 -- 172.21.15.61:0/1985619245 >> 172.21.15.61:0/1985619245 conn(0x7f3cdc0f6590 msgr2=0x7f3cdc0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 -- 172.21.15.61:0/1985619245 shutdown_connections 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.461+0000 7f3ce10ba700 1 -- 172.21.15.61:0/1985619245 wait complete. 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3ce10ba700 1 Processor -- start 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3ce10ba700 1 -- start start 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3ce10ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3ce10ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc10d3a0 con 0x7f3cdc027610 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3cdad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3cdad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46724/0 (socket says 172.21.15.61:46724) 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.462+0000 7f3cdad9d700 1 -- 172.21.15.61:0/3954057901 learned_addr learned my addr 172.21.15.61:0/3954057901 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cdad9d700 1 -- 172.21.15.61:0/3954057901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc00b200 con 0x7f3cdc027610 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cdad9d700 1 --2- 172.21.15.61:0/3954057901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ccc000c00 tx=0x7f3ccc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc01b410 con 0x7f3cdc027610 2023-11-16T14:00:54.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3ccc01b9d0 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc10d5a0 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc01a620 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc10e1b0 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f3ccc02b070 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.463+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3ccc0249b0 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.464+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc0591e0 con 0x7f3cdc027610 2023-11-16T14:00:54.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.467+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3ccc01f020 con 0x7f3cdc027610 2023-11-16T14:00:54.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.543+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3cdc10efe0 con 0x7f3cdc027610 2023-11-16T14:00:54.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.543+0000 7f3cc37fe700 1 -- 172.21.15.61:0/3954057901 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f3ccc016070 con 0x7f3cdc027610 2023-11-16T14:00:54.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "71d22a3a-8488-11ee-8dbe-212e2dc638e7", 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-11-16T14:00:54.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:54.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T14:00:54.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": false, 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-11-16T14:00:54.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "nfs", 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-11-16T14:00:49.168022+0000", 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 msgr2=0x7f3cdc10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:54.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 --2- 172.21.15.61:0/3954057901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ccc000c00 tx=0x7f3ccc006ee0).stop 2023-11-16T14:00:54.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 shutdown_connections 2023-11-16T14:00:54.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 --2- 172.21.15.61:0/3954057901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc027610 0x7f3cdc10ceb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:54.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 >> 172.21.15.61:0/3954057901 conn(0x7f3cdc0f6590 msgr2=0x7f3cdc0f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:54.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 shutdown_connections 2023-11-16T14:00:54.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:54.544+0000 7f3ce10ba700 1 -- 172.21.15.61:0/3954057901 wait complete. 2023-11-16T14:00:55.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:54 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:54.544415+0000 mon.smithi061 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.61:0/3954057901' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T14:00:55.875 INFO:teuthology.orchestra.run.smithi061.stderr:mgr not available, waiting (1/15)... 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.231+0000 7ff9cfb49700 1 Processor -- start 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cfb49700 1 -- start start 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cfb49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cfb49700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80fccd0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cd8e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cd8e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40954/0 (socket says 172.21.15.61:40954) 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.232+0000 7ff9cd8e5700 1 -- 172.21.15.61:0/3607773854 learned_addr learned my addr 172.21.15.61:0/3607773854 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.233+0000 7ff9cd8e5700 1 -- 172.21.15.61:0/3607773854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c80fce10 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.233+0000 7ff9cd8e5700 1 --2- 172.21.15.61:0/3607773854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff9c400b870 tx=0x7ff9c4006f70).ready entity=mon.0 client_cookie=abe58076a1168344 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.233+0000 7ff9cc8e3700 1 -- 172.21.15.61:0/3607773854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c400da20 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.233+0000 7ff9cc8e3700 1 -- 172.21.15.61:0/3607773854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff9c4010470 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.233+0000 7ff9cc8e3700 1 -- 172.21.15.61:0/3607773854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c400f5b0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 -- 172.21.15.61:0/3607773854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 msgr2=0x7ff9c80fc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 --2- 172.21.15.61:0/3607773854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff9c400b870 tx=0x7ff9c4006f70).stop 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 -- 172.21.15.61:0/3607773854 shutdown_connections 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 --2- 172.21.15.61:0/3607773854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c80fc7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 -- 172.21.15.61:0/3607773854 >> 172.21.15.61:0/3607773854 conn(0x7ff9c80f5ce0 msgr2=0x7ff9c80f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 -- 172.21.15.61:0/3607773854 shutdown_connections 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.234+0000 7ff9cfb49700 1 -- 172.21.15.61:0/3607773854 wait complete. 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cfb49700 1 Processor -- start 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cfb49700 1 -- start start 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cfb49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cfb49700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8108f60 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cd8e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cd8e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40956/0 (socket says 172.21.15.61:40956) 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cd8e5700 1 -- 172.21.15.61:0/1887979454 learned_addr learned my addr 172.21.15.61:0/1887979454 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.235+0000 7ff9cd8e5700 1 -- 172.21.15.61:0/1887979454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c400b480 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9cd8e5700 1 --2- 172.21.15.61:0/1887979454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff9c4010d50 tx=0x7ff9c4006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c400f3d0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff9c4006c10 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c4019bb0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8109160 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c81095f0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.236+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7ff9c4016030 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.237+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff9c402dbd0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.237+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c8045bb0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.239+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff9c4014070 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.311 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.309+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff9c80fa9f0 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.309+0000 7ff9b67fc700 1 -- 172.21.15.61:0/1887979454 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7ff9c4019e20 con 0x7ff9c80fc3c0 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "71d22a3a-8488-11ee-8dbe-212e2dc638e7", 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T14:00:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T14:00:58.313 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-11-16T14:00:58.313 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 5, 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T14:00:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T14:00:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T14:00:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": false, 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "nfs", 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:00:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-11-16T14:00:49.168022+0000", 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 msgr2=0x7ff9c8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 --2- 172.21.15.61:0/1887979454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff9c4010d50 tx=0x7ff9c4006f70).stop 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 shutdown_connections 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 --2- 172.21.15.61:0/1887979454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9c80fc3c0 0x7ff9c8108a70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:00:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 >> 172.21.15.61:0/1887979454 conn(0x7ff9c80f5ce0 msgr2=0x7ff9c80ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:00:58.322 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 shutdown_connections 2023-11-16T14:00:58.322 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:00:58.310+0000 7ff9cfb49700 1 -- 172.21.15.61:0/1887979454 wait complete. 2023-11-16T14:00:58.644 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:58 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:58.309294+0000 mon.smithi061 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.61:0/1887979454' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T14:00:58.657 INFO:teuthology.orchestra.run.smithi061.stderr:mgr not available, waiting (2/15)... 2023-11-16T14:01:00.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:59.171461+0000 mon.smithi061 (mon.0) 9 : cluster [INF] Activating manager daemon smithi061.zcjido 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:59.174098+0000 mon.smithi061 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi061.zcjido(active, starting, since 0.00271499s) 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.175160+0000 mon.smithi061 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.175281+0000 mon.smithi061 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.175388+0000 mon.smithi061 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.175499+0000 mon.smithi061 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.175787+0000 mon.smithi061 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mgr metadata", "who": "smithi061.zcjido", "id": "smithi061.zcjido"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:00:59.189338+0000 mon.smithi061 (mon.0) 16 : cluster [INF] Manager daemon smithi061.zcjido is now available 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.197315+0000 mon.smithi061 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.201814+0000 mon.smithi061 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.202972+0000 mon.smithi061 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:01:00.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.207703+0000 mon.smithi061 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:00.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:00:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:00:59.209567+0000 mon.smithi061 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.61:0/3894093478' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:01.139 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.135+0000 7fe966339700 1 Processor -- start 2023-11-16T14:01:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe966339700 1 -- start start 2023-11-16T14:01:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe966339700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe966339700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600fb710 con 0x7fe9600fae00 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41032/0 (socket says 172.21.15.61:41032) 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe95ffff700 1 -- 172.21.15.61:0/1214192578 learned_addr learned my addr 172.21.15.61:0/1214192578 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.136+0000 7fe95ffff700 1 -- 172.21.15.61:0/1214192578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600fb850 con 0x7fe9600fae00 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe95ffff700 1 --2- 172.21.15.61:0/1214192578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe95000b5f0 tx=0x7fe950006ee0).ready entity=mon.0 client_cookie=bec85fe82f028398 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe95effd700 1 -- 172.21.15.61:0/1214192578 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95000d600 con 0x7fe9600fae00 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe95effd700 1 -- 172.21.15.61:0/1214192578 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe95000dbc0 con 0x7fe9600fae00 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe95effd700 1 -- 172.21.15.61:0/1214192578 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe950011b90 con 0x7fe9600fae00 2023-11-16T14:01:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 -- 172.21.15.61:0/1214192578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 msgr2=0x7fe9600fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 --2- 172.21.15.61:0/1214192578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe95000b5f0 tx=0x7fe950006ee0).stop 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 -- 172.21.15.61:0/1214192578 shutdown_connections 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 --2- 172.21.15.61:0/1214192578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe9600fb220 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 -- 172.21.15.61:0/1214192578 >> 172.21.15.61:0/1214192578 conn(0x7fe9600f6590 msgr2=0x7fe9600f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.137+0000 7fe966339700 1 -- 172.21.15.61:0/1214192578 shutdown_connections 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.138+0000 7fe966339700 1 -- 172.21.15.61:0/1214192578 wait complete. 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.138+0000 7fe966339700 1 Processor -- start 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.138+0000 7fe966339700 1 -- start start 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.138+0000 7fe966339700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.138+0000 7fe966339700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960027650 con 0x7fe9600fae00 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41036/0 (socket says 172.21.15.61:41036) 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95ffff700 1 -- 172.21.15.61:0/1203062890 learned_addr learned my addr 172.21.15.61:0/1203062890 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95ffff700 1 -- 172.21.15.61:0/1203062890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95000b200 con 0x7fe9600fae00 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95ffff700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe950000c00 tx=0x7fe950006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95000de00 con 0x7fe9600fae00 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe95001b870 con 0x7fe9600fae00 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe960027850 con 0x7fe9600fae00 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95001a670 con 0x7fe9600fae00 2023-11-16T14:01:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.139+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96010e060 con 0x7fe9600fae00 2023-11-16T14:01:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.140+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 3) v1 ==== 37004+0+0 (secure 0 0 0) 0x7fe95002b070 con 0x7fe9600fae00 2023-11-16T14:01:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.140+0000 7fe95cff9700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 0x7fe94c0333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.140+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe950042ab0 con 0x7fe9600fae00 2023-11-16T14:01:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.140+0000 7fe95f7fe700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 0x7fe94c0333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.141+0000 7fe95f7fe700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 0x7fe94c0333a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe9540096a0 tx=0x7fe954006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:01.147 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.141+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe960045bb0 con 0x7fe9600fae00 2023-11-16T14:01:01.148 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.145+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe95001f080 con 0x7fe9600fae00 2023-11-16T14:01:01.375 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.366+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe96006b250 con 0x7fe9600fae00 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.367+0000 7fe95cff9700 1 -- 172.21.15.61:0/1203062890 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe950016070 con 0x7fe9600fae00 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "71d22a3a-8488-11ee-8dbe-212e2dc638e7", 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T14:01:01.376 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T14:01:01.377 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-11-16T14:01:01.377 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 8, 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.378 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T14:01:01.379 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T14:01:01.380 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "nfs", 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.381 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-11-16T14:00:49.168022+0000", 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 msgr2=0x7fe94c0333a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 0x7fe94c0333a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe9540096a0 tx=0x7fe954006b40).stop 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 msgr2=0x7fe96010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:01.382 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe950000c00 tx=0x7fe950006ee0).stop 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 shutdown_connections 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7fe94c030ee0 0x7fe94c0333a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 --2- 172.21.15.61:0/1203062890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9600fae00 0x7fe96010d930 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 >> 172.21.15.61:0/1203062890 conn(0x7fe9600f6590 msgr2=0x7fe9600f80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 shutdown_connections 2023-11-16T14:01:01.383 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:01.370+0000 7fe966339700 1 -- 172.21.15.61:0/1203062890 wait complete. 2023-11-16T14:01:01.395 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:00.177507+0000 mon.smithi061 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi061.zcjido(active, since 1.00612s) 2023-11-16T14:01:02.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:02 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:01.367952+0000 mon.smithi061 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.61:0/1203062890' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T14:01:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:mgr is available 2023-11-16T14:01:03.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:02.177437+0000 mon.smithi061 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi061.zcjido(active, since 3s) 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2557114700 1 Processor -- start 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2557114700 1 -- start start 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2557114700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2557114700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25500fc980 con 0x7f25500275f0 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2554eb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2554eb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41048/0 (socket says 172.21.15.61:41048) 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.710+0000 7f2554eb0700 1 -- 172.21.15.61:0/512227984 learned_addr learned my addr 172.21.15.61:0/512227984 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.711+0000 7f2554eb0700 1 -- 172.21.15.61:0/512227984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550101480 con 0x7f25500275f0 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.711+0000 7f2554eb0700 1 --2- 172.21.15.61:0/512227984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f254400b5f0 tx=0x7f2544006ee0).ready entity=mon.0 client_cookie=9b7764cd091a7545 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.711+0000 7f254f7fe700 1 -- 172.21.15.61:0/512227984 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254400d600 con 0x7f25500275f0 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.711+0000 7f254f7fe700 1 -- 172.21.15.61:0/512227984 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f254400dbc0 con 0x7f25500275f0 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.711+0000 7f254f7fe700 1 -- 172.21.15.61:0/512227984 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2544011b90 con 0x7f25500275f0 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 -- 172.21.15.61:0/512227984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 msgr2=0x7f2550100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 --2- 172.21.15.61:0/512227984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f254400b5f0 tx=0x7f2544006ee0).stop 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 -- 172.21.15.61:0/512227984 shutdown_connections 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 --2- 172.21.15.61:0/512227984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f2550100f90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 -- 172.21.15.61:0/512227984 >> 172.21.15.61:0/512227984 conn(0x7f25500f6590 msgr2=0x7f25500f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 -- 172.21.15.61:0/512227984 shutdown_connections 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 -- 172.21.15.61:0/512227984 wait complete. 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.712+0000 7f2557114700 1 Processor -- start 2023-11-16T14:01:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2557114700 1 -- start start 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2557114700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2557114700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255010d3b0 con 0x7f25500275f0 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2554eb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2554eb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41050/0 (socket says 172.21.15.61:41050) 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.713+0000 7f2554eb0700 1 -- 172.21.15.61:0/760210066 learned_addr learned my addr 172.21.15.61:0/760210066 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f2554eb0700 1 -- 172.21.15.61:0/760210066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254400b200 con 0x7f25500275f0 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f2554eb0700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2544006800 tx=0x7f2544006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254400d9c0 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f254401b870 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255010d5b0 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254401a740 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.714+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255010e1c0 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f254402b070 con 0x7f25500275f0 2023-11-16T14:01:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f254d7fa700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 0x7f2534033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2544042bc0 con 0x7f25500275f0 2023-11-16T14:01:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f254ffff700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 0x7f2534033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255010da40 con 0x7f25500275f0 2023-11-16T14:01:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.715+0000 7f254ffff700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 0x7f2534033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f253c0096a0 tx=0x7f253c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.720+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f255010da40 con 0x7f25500275f0 2023-11-16T14:01:04.887 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.885+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2550045bb0 con 0x7f25500275f0 2023-11-16T14:01:04.892 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.885+0000 7f254d7fa700 1 -- 172.21.15.61:0/760210066 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f2544033db0 con 0x7f25500275f0 2023-11-16T14:01:04.892 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.892 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [global] 2023-11-16T14:01:04.892 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: fsid = 71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:01:04.892 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: log_to_journald = false 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.893 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [mgr] 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [osd] 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-16T14:01:04.894 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 msgr2=0x7f2534033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 0x7f2534033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f253c0096a0 tx=0x7f253c006b40).stop 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 msgr2=0x7f255010cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2544006800 tx=0x7f2544006ee0).stop 2023-11-16T14:01:04.895 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 shutdown_connections 2023-11-16T14:01:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f2534030fa0 0x7f2534033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 --2- 172.21.15.61:0/760210066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25500275f0 0x7f255010cec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 >> 172.21.15.61:0/760210066 conn(0x7f25500f6590 msgr2=0x7f25500f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 shutdown_connections 2023-11-16T14:01:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:04.888+0000 7f2557114700 1 -- 172.21.15.61:0/760210066 wait complete. 2023-11-16T14:01:05.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:04 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:04.886369+0000 mon.smithi061 (mon.0) 25 : audit [INF] from='client.? 172.21.15.61:0/760210066' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T14:01:06.966 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling cephadm module... 2023-11-16T14:01:08.141 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.127+0000 7f1cb53b8700 1 Processor -- start 2023-11-16T14:01:08.141 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.127+0000 7f1cb53b8700 1 -- start start 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.127+0000 7f1cb53b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.127+0000 7f1cb53b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00fc800 con 0x7f1cb00fbef0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56802/0 (socket says 172.21.15.61:56802) 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1caeffd700 1 -- 172.21.15.61:0/231306291 learned_addr learned my addr 172.21.15.61:0/231306291 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1caeffd700 1 -- 172.21.15.61:0/231306291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00fc940 con 0x7f1cb00fbef0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1caeffd700 1 --2- 172.21.15.61:0/231306291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f1c9800b5f0 tx=0x7f1c98006ee0).ready entity=mon.0 client_cookie=90956ee142f1d189 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1cadffb700 1 -- 172.21.15.61:0/231306291 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c9800d600 con 0x7f1cb00fbef0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1cadffb700 1 -- 172.21.15.61:0/231306291 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1c9800dbc0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.128+0000 7f1cadffb700 1 -- 172.21.15.61:0/231306291 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c98011b90 con 0x7f1cb00fbef0 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 -- 172.21.15.61:0/231306291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 msgr2=0x7f1cb00fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:08.142 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/231306291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f1c9800b5f0 tx=0x7f1c98006ee0).stop 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 -- 172.21.15.61:0/231306291 shutdown_connections 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/231306291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb00fc310 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 -- 172.21.15.61:0/231306291 >> 172.21.15.61:0/231306291 conn(0x7f1cb00f7640 msgr2=0x7f1cb00f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 -- 172.21.15.61:0/231306291 shutdown_connections 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 -- 172.21.15.61:0/231306291 wait complete. 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.129+0000 7f1cb53b8700 1 Processor -- start 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1cb53b8700 1 -- start start 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1cb53b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1cb53b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010b240 con 0x7f1cb00fbef0 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56804/0 (socket says 172.21.15.61:56804) 2023-11-16T14:01:08.143 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1caeffd700 1 -- 172.21.15.61:0/3125709438 learned_addr learned my addr 172.21.15.61:0/3125709438 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1caeffd700 1 -- 172.21.15.61:0/3125709438 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9800b200 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1caeffd700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1c98006800 tx=0x7f1c98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c9800de50 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1c9801b870 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb010b440 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.130+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c9801a7a0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.131+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb010bff0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.131+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f1c9800f070 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.131+0000 7f1ca77fe700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 0x7f1c90033400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.131+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1c98042ba0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.131+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb00591e0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.132+0000 7f1cae7fc700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 0x7f1c90033400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:08.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.132+0000 7f1cae7fc700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 0x7f1c90033400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1ca00096a0 tx=0x7f1ca0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:08.145 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.136+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c9802b8e0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.344 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.341+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1cb010ce20 con 0x7f1cb00fbef0 2023-11-16T14:01:08.514 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.511+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f1c9801b9e0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.515 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.512+0000 7f1ca77fe700 1 -- 172.21.15.61:0/3125709438 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1c980239e0 con 0x7f1cb00fbef0 2023-11-16T14:01:08.517 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 msgr2=0x7f1c90033400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:08.517 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 0x7f1c90033400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1ca00096a0 tx=0x7f1ca0006b40).stop 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 msgr2=0x7f1cb010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1c98006800 tx=0x7f1c98006ee0).stop 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 shutdown_connections 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f1c90030f40 0x7f1c90033400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 --2- 172.21.15.61:0/3125709438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cb00fbef0 0x7f1cb010ad50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.514+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 >> 172.21.15.61:0/3125709438 conn(0x7f1cb00f7640 msgr2=0x7f1cb00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.515+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 shutdown_connections 2023-11-16T14:01:08.518 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:08.515+0000 7f1cb53b8700 1 -- 172.21.15.61:0/3125709438 wait complete. 2023-11-16T14:01:08.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:08.343199+0000 mon.smithi061 (mon.0) 26 : audit [INF] from='client.? 172.21.15.61:0/3125709438' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T14:01:09.830 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:08.513232+0000 mon.smithi061 (mon.0) 27 : audit [INF] from='client.? 172.21.15.61:0/3125709438' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T14:01:09.830 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:08.513284+0000 mon.smithi061 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi061.zcjido(active, since 9s) 2023-11-16T14:01:11.074 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186ffff700 1 Processor -- start 2023-11-16T14:01:11.074 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186ffff700 1 -- start start 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700fee70 con 0x7f187006ade0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56830/0 (socket says 172.21.15.61:56830) 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.068+0000 7f186effd700 1 -- 172.21.15.61:0/1176963876 learned_addr learned my addr 172.21.15.61:0/1176963876 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.069+0000 7f186effd700 1 -- 172.21.15.61:0/1176963876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700fefb0 con 0x7f187006ade0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.069+0000 7f186effd700 1 --2- 172.21.15.61:0/1176963876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f186000b5f0 tx=0x7f1860006ee0).ready entity=mon.0 client_cookie=f9182f01610f2c94 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.069+0000 7f186dffb700 1 -- 172.21.15.61:0/1176963876 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f186000d600 con 0x7f187006ade0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.069+0000 7f186dffb700 1 -- 172.21.15.61:0/1176963876 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f186000dbc0 con 0x7f187006ade0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.069+0000 7f186dffb700 1 -- 172.21.15.61:0/1176963876 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1860011b90 con 0x7f187006ade0 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 -- 172.21.15.61:0/1176963876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 msgr2=0x7f18700fe980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 --2- 172.21.15.61:0/1176963876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f186000b5f0 tx=0x7f1860006ee0).stop 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 -- 172.21.15.61:0/1176963876 shutdown_connections 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 --2- 172.21.15.61:0/1176963876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f18700fe980 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 -- 172.21.15.61:0/1176963876 >> 172.21.15.61:0/1176963876 conn(0x7f18700f63c0 msgr2=0x7f18700f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 -- 172.21.15.61:0/1176963876 shutdown_connections 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.070+0000 7f186ffff700 1 -- 172.21.15.61:0/1176963876 wait complete. 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186ffff700 1 Processor -- start 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186ffff700 1 -- start start 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187010afb0 con 0x7f187006ade0 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56832/0 (socket says 172.21.15.61:56832) 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186effd700 1 -- 172.21.15.61:0/4110339435 learned_addr learned my addr 172.21.15.61:0/4110339435 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186effd700 1 -- 172.21.15.61:0/4110339435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186000b200 con 0x7f187006ade0 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.071+0000 7f186effd700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1860000c00 tx=0x7f1860006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.072+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f186001b410 con 0x7f187006ade0 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.072+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f186001b9d0 con 0x7f187006ade0 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.072+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187010b1b0 con 0x7f187006ade0 2023-11-16T14:01:11.077 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.072+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f186001a620 con 0x7f187006ade0 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.072+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187010b640 con 0x7f187006ade0 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f186000f070 con 0x7f187006ade0 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f18577fe700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 0x7f185c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f186001be40 con 0x7f187006ade0 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f186e7fc700 1 -- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 msgr2=0x7f185c0334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:11.078 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f186e7fc700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 0x7f185c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T14:01:11.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.073+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1870045bb0 con 0x7f187006ade0 2023-11-16T14:01:11.080 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.078+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f186001f070 con 0x7f187006ade0 2023-11-16T14:01:11.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.274+0000 7f186e7fc700 1 -- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 msgr2=0x7f185c0334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:11.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.274+0000 7f186e7fc700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 0x7f185c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T14:01:11.283 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.280+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f18701049e0 con 0x7f187006ade0 2023-11-16T14:01:11.283 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.280+0000 7f18577fe700 1 -- 172.21.15.61:0/4110339435 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f186001b570 con 0x7f187006ade0 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "active_name": "smithi061.zcjido", 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-16T14:01:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 msgr2=0x7f185c0334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 0x7f185c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 msgr2=0x7f187010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1860000c00 tx=0x7f1860006ee0).stop 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 shutdown_connections 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f185c030ff0 0x7f185c0334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 --2- 172.21.15.61:0/4110339435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f187006ade0 0x7f187010aac0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 >> 172.21.15.61:0/4110339435 conn(0x7f18700f63c0 msgr2=0x7f18700f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:11.287 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 shutdown_connections 2023-11-16T14:01:11.287 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.282+0000 7f186ffff700 1 -- 172.21.15.61:0/4110339435 wait complete. 2023-11-16T14:01:11.592 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:11.281688+0000 mon.smithi061 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.61:0/4110339435' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T14:01:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for the mgr to restart... 2023-11-16T14:01:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr epoch 5... 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.997+0000 7f93a2197700 1 Processor -- start 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.997+0000 7f93a2197700 1 -- start start 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.997+0000 7f93a2197700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.997+0000 7f93a2197700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0fc810 con 0x7f939c0fbf00 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56844/0 (socket says 172.21.15.61:56844) 2023-11-16T14:01:12.001 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939b7fe700 1 -- 172.21.15.61:0/2046326873 learned_addr learned my addr 172.21.15.61:0/2046326873 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:12.002 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939b7fe700 1 -- 172.21.15.61:0/2046326873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c0fc950 con 0x7f939c0fbf00 2023-11-16T14:01:12.002 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939b7fe700 1 --2- 172.21.15.61:0/2046326873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f938c00b5f0 tx=0x7f938c006ee0).ready entity=mon.0 client_cookie=48765cd783ba6db8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:12.002 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939a7fc700 1 -- 172.21.15.61:0/2046326873 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938c00d600 con 0x7f939c0fbf00 2023-11-16T14:01:12.002 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939a7fc700 1 -- 172.21.15.61:0/2046326873 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f938c00dbc0 con 0x7f939c0fbf00 2023-11-16T14:01:12.002 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.998+0000 7f939a7fc700 1 -- 172.21.15.61:0/2046326873 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938c011b90 con 0x7f939c0fbf00 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 -- 172.21.15.61:0/2046326873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 msgr2=0x7f939c0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 --2- 172.21.15.61:0/2046326873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f938c00b5f0 tx=0x7f938c006ee0).stop 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 -- 172.21.15.61:0/2046326873 shutdown_connections 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 --2- 172.21.15.61:0/2046326873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c0fc320 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 -- 172.21.15.61:0/2046326873 >> 172.21.15.61:0/2046326873 conn(0x7f939c0f7650 msgr2=0x7f939c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 -- 172.21.15.61:0/2046326873 shutdown_connections 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:11.999+0000 7f93a2197700 1 -- 172.21.15.61:0/2046326873 wait complete. 2023-11-16T14:01:12.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f93a2197700 1 Processor -- start 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f93a2197700 1 -- start start 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f93a2197700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f93a2197700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c10b0f0 con 0x7f939c0fbf00 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f939b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f939b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56858/0 (socket says 172.21.15.61:56858) 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f939b7fe700 1 -- 172.21.15.61:0/4080825790 learned_addr learned my addr 172.21.15.61:0/4080825790 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.000+0000 7f939b7fe700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c00b200 con 0x7f939c0fbf00 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f939b7fe700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f938c006800 tx=0x7f938c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938c01b410 con 0x7f939c0fbf00 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f938c01b9d0 con 0x7f939c0fbf00 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c10b2f0 con 0x7f939c0fbf00 2023-11-16T14:01:12.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938c01a840 con 0x7f939c0fbf00 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.001+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c10b780 con 0x7f939c0fbf00 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f938c00f070 con 0x7f939c0fbf00 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f9383fff700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f939c0591e0 con 0x7f9388030ff0 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f938c042b40 con 0x7f939c0fbf00 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f939affd700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 msgr2=0x7f93880334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:12.007 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.002+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T14:01:12.205 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.203+0000 7f939affd700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 msgr2=0x7f93880334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:12.206 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.203+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T14:01:12.606 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.604+0000 7f939affd700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 msgr2=0x7f93880334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:12.607 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:12.604+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-16T14:01:13.407 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:13.405+0000 7f939affd700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 msgr2=0x7f93880334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1180654700 2023-11-16T14:01:13.408 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:13.405+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-16T14:01:13.910 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:13.906+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7f938c01bb40 con 0x7f939c0fbf00 2023-11-16T14:01:13.910 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:13.906+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 msgr2=0x7f93880334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T14:01:13.910 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:13.906+0000 7f9383fff700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:14.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:13.852900+0000 mon.smithi061 (mon.0) 30 : cluster [INF] Active manager daemon smithi061.zcjido restarted 2023-11-16T14:01:14.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:13.853290+0000 mon.smithi061 (mon.0) 31 : cluster [INF] Activating manager daemon smithi061.zcjido 2023-11-16T14:01:14.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:13.857441+0000 mon.smithi061 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-16T14:01:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.916+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f938c044070 con 0x7f939c0fbf00 2023-11-16T14:01:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.916+0000 7f9383fff700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 0x7f9388036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.916+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f939c0591e0 con 0x7f9388034850 2023-11-16T14:01:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.917+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 0x7f9388036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.917+0000 7f939affd700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 0x7f9388036c40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f93900036d0 tx=0x7f9390006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:14.922 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.918+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f939c0591e0 con 0x7f9388034850 2023-11-16T14:01:14.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.925+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f939c045bb0 con 0x7f9388034850 2023-11-16T14:01:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.925+0000 7f9383fff700 1 -- 172.21.15.61:0/4080825790 <== mgr.14116 v2:172.21.15.61:6800/1639220556 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f939c045bb0 con 0x7f9388034850 2023-11-16T14:01:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:01:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-16T14:01:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "initialized": true 2023-11-16T14:01:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 msgr2=0x7f9388036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 0x7f9388036c40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f93900036d0 tx=0x7f9390006d20).stop 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 msgr2=0x7f939c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f938c006800 tx=0x7f938c006ee0).stop 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 shutdown_connections 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f939c0fbf00 0x7f939c10ac00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9388034850 0x7f9388036c40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 --2- 172.21.15.61:0/4080825790 >> [v2:172.21.15.61:6800/1180654700,v1:172.21.15.61:6801/1180654700] conn(0x7f9388030ff0 0x7f93880334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 >> 172.21.15.61:0/4080825790 conn(0x7f939c0f7650 msgr2=0x7f939c0f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 shutdown_connections 2023-11-16T14:01:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:14.926+0000 7f93a2197700 1 -- 172.21.15.61:0/4080825790 wait complete. 2023-11-16T14:01:15.250 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:13.908519+0000 mon.smithi061 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi061.zcjido(active, starting, since 0.0553078s) 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.911239+0000 mon.smithi061 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.911415+0000 mon.smithi061 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mgr metadata", "who": "smithi061.zcjido", "id": "smithi061.zcjido"}]: dispatch 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.911575+0000 mon.smithi061 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.911751+0000 mon.smithi061 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.911870+0000 mon.smithi061 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T14:01:15.251 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:13.925482+0000 mon.smithi061 (mon.0) 39 : cluster [INF] Manager daemon smithi061.zcjido is now available 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.939109+0000 mon.smithi061 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.941792+0000 mon.smithi061 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:13.944465+0000 mon.smithi061 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.038320+0000 mon.smithi061 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.038988+0000 mon.smithi061 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.041505+0000 mon.smithi061 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.045652+0000 mon.smithi061 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.053270+0000 mon.smithi061 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:01:15.252 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.054448+0000 mon.smithi061 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:01:15.256 INFO:teuthology.orchestra.run.smithi061.stderr:mgr epoch 5 is available 2023-11-16T14:01:15.256 INFO:teuthology.orchestra.run.smithi061.stderr:Setting orchestrator backend to cephadm... 2023-11-16T14:01:15.593 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe867cef700 1 Processor -- start 2023-11-16T14:01:15.593 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe867cef700 1 -- start start 2023-11-16T14:01:15.593 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe867cef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:15.594 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe867cef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600fcd90 con 0x7fe8600fc4a0 2023-11-16T14:01:15.594 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe865a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:15.594 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe865a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56904/0 (socket says 172.21.15.61:56904) 2023-11-16T14:01:15.594 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.589+0000 7fe865a8b700 1 -- 172.21.15.61:0/3655299640 learned_addr learned my addr 172.21.15.61:0/3655299640 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:15.595 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe865a8b700 1 -- 172.21.15.61:0/3655299640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8600fced0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.595 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe865a8b700 1 --2- 172.21.15.61:0/3655299640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe85c00b5f0 tx=0x7fe85c006ee0).ready entity=mon.0 client_cookie=85d2233836c62aff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:15.595 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe864a89700 1 -- 172.21.15.61:0/3655299640 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c00d600 con 0x7fe8600fc4a0 2023-11-16T14:01:15.595 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe864a89700 1 -- 172.21.15.61:0/3655299640 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe85c00dbc0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.596 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe864a89700 1 -- 172.21.15.61:0/3655299640 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c011b90 con 0x7fe8600fc4a0 2023-11-16T14:01:15.596 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe867cef700 1 -- 172.21.15.61:0/3655299640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 msgr2=0x7fe8600fc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe867cef700 1 --2- 172.21.15.61:0/3655299640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe85c00b5f0 tx=0x7fe85c006ee0).stop 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe867cef700 1 -- 172.21.15.61:0/3655299640 shutdown_connections 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe867cef700 1 --2- 172.21.15.61:0/3655299640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe8600fc8a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.590+0000 7fe867cef700 1 -- 172.21.15.61:0/3655299640 >> 172.21.15.61:0/3655299640 conn(0x7fe8600f5d10 msgr2=0x7fe8600f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 -- 172.21.15.61:0/3655299640 shutdown_connections 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 -- 172.21.15.61:0/3655299640 wait complete. 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 Processor -- start 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 -- start start 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:15.599 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.591+0000 7fe867cef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86010d3c0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe865a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe865a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56920/0 (socket says 172.21.15.61:56920) 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe865a8b700 1 -- 172.21.15.61:0/384759669 learned_addr learned my addr 172.21.15.61:0/384759669 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe865a8b700 1 -- 172.21.15.61:0/384759669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c00b200 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe865a8b700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe85c006800 tx=0x7fe85c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c00de50 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fe85c01b870 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86010d5c0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c01a7a0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.592+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86010e170 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.593+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7fe85c00f070 con 0x7fe8600fc4a0 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.593+0000 7fe84e7fc700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 0x7fe8480333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:15.600 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.593+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fe85c0434d0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.601 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.593+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8600591e0 con 0x7fe8600fc4a0 2023-11-16T14:01:15.601 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.593+0000 7fe86528a700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 0x7fe8480333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:15.601 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.594+0000 7fe86528a700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 0x7fe8480333c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe8500096a0 tx=0x7fe850006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:15.601 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.598+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe85c016030 con 0x7fe8600fc4a0 2023-11-16T14:01:15.772 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.770+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fe86010e470 con 0x7fe848030f00 2023-11-16T14:01:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.775+0000 7fe84e7fc700 1 -- 172.21.15.61:0/384759669 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe86010e470 con 0x7fe848030f00 2023-11-16T14:01:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 msgr2=0x7fe8480333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 0x7fe8480333c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe8500096a0 tx=0x7fe850006b40).stop 2023-11-16T14:01:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 msgr2=0x7fe86010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe85c006800 tx=0x7fe85c006ee0).stop 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 shutdown_connections 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fe848030f00 0x7fe8480333c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 --2- 172.21.15.61:0/384759669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8600fc4a0 0x7fe86010ced0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.777+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 >> 172.21.15.61:0/384759669 conn(0x7fe8600f5d10 msgr2=0x7fe8600f77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.778+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 shutdown_connections 2023-11-16T14:01:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:15.778+0000 7fe867cef700 1 -- 172.21.15.61:0/384759669 wait complete. 2023-11-16T14:01:16.218 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.919168+0000 mgr.smithi061.zcjido (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T14:01:16.218 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:14.919768+0000 mon.smithi061 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi061.zcjido(active, since 1.06655s) 2023-11-16T14:01:16.219 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:14.926493+0000 mgr.smithi061.zcjido (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T14:01:16.219 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:15.775786+0000 mon.smithi061 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:16.219 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:15.779488+0000 mon.smithi061 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:16.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.450+0000 7f39cf4fe700 1 Processor -- start 2023-11-16T14:01:16.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.450+0000 7f39cf4fe700 1 -- start start 2023-11-16T14:01:16.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.450+0000 7f39cf4fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:16.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.450+0000 7f39cf4fe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80fc810 con 0x7f39c80fbf00 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39cd29a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39cd29a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56932/0 (socket says 172.21.15.61:56932) 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39cd29a700 1 -- 172.21.15.61:0/4122551519 learned_addr learned my addr 172.21.15.61:0/4122551519 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39cd29a700 1 -- 172.21.15.61:0/4122551519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c80fc950 con 0x7f39c80fbf00 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39cd29a700 1 --2- 172.21.15.61:0/4122551519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f39b800b5f0 tx=0x7f39b8006ee0).ready entity=mon.0 client_cookie=4760ccc2aac40789 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39b7fff700 1 -- 172.21.15.61:0/4122551519 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39b800d600 con 0x7f39c80fbf00 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39b7fff700 1 -- 172.21.15.61:0/4122551519 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f39b800dbc0 con 0x7f39c80fbf00 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.451+0000 7f39b7fff700 1 -- 172.21.15.61:0/4122551519 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39b8011b90 con 0x7f39c80fbf00 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- 172.21.15.61:0/4122551519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 msgr2=0x7f39c80fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:16.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/4122551519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f39b800b5f0 tx=0x7f39b8006ee0).stop 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- 172.21.15.61:0/4122551519 shutdown_connections 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/4122551519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c80fc320 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- 172.21.15.61:0/4122551519 >> 172.21.15.61:0/4122551519 conn(0x7f39c80f7650 msgr2=0x7f39c80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- 172.21.15.61:0/4122551519 shutdown_connections 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- 172.21.15.61:0/4122551519 wait complete. 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 Processor -- start 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.452+0000 7f39cf4fe700 1 -- start start 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cf4fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:16.458 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cf4fe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80700d0 con 0x7f39c80fbf00 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cd29a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cd29a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56946/0 (socket says 172.21.15.61:56946) 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cd29a700 1 -- 172.21.15.61:0/3200569668 learned_addr learned my addr 172.21.15.61:0/3200569668 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cd29a700 1 -- 172.21.15.61:0/3200569668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b800b200 con 0x7f39c80fbf00 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cd29a700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f39b8000c00 tx=0x7f39b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39b801b570 con 0x7f39c80fbf00 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c8070270 con 0x7f39c80fbf00 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f39b801bb30 con 0x7f39c80fbf00 2023-11-16T14:01:16.459 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.453+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39b801a740 con 0x7f39c80fbf00 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.454+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8070700 con 0x7f39c80fbf00 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.454+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f39b802b070 con 0x7f39c80fbf00 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.454+0000 7f39b5ffb700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 0x7f39b00333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.454+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c8045bb0 con 0x7f39c80fbf00 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.455+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f39b8042c30 con 0x7f39c80fbf00 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.455+0000 7f39cca99700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 0x7f39b00333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:16.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.455+0000 7f39cca99700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 0x7f39b00333c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f39c40096a0 tx=0x7f39c4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:16.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.459+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f39b8016080 con 0x7f39c80fbf00 2023-11-16T14:01:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.620+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f39c80fddc0 con 0x7f39b0030f00 2023-11-16T14:01:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.621+0000 7f39b5ffb700 1 -- 172.21.15.61:0/3200569668 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f39c80fddc0 con 0x7f39b0030f00 2023-11-16T14:01:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: value unchanged 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 msgr2=0x7f39b00333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 0x7f39b00333c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f39c40096a0 tx=0x7f39c4006b40).stop 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 msgr2=0x7f39c8071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f39b8000c00 tx=0x7f39b8006ee0).stop 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 shutdown_connections 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f39b0030f00 0x7f39b00333c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 --2- 172.21.15.61:0/3200569668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39c80fbf00 0x7f39c8071c50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 >> 172.21.15.61:0/3200569668 conn(0x7f39c80f7650 msgr2=0x7f39c80f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.622+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 shutdown_connections 2023-11-16T14:01:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:16.623+0000 7f39cf4fe700 1 -- 172.21.15.61:0/3200569668 wait complete. 2023-11-16T14:01:16.947 INFO:teuthology.orchestra.run.smithi061.stderr:Generating ssh key... 2023-11-16T14:01:17.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:16 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:15.771844+0000 mgr.smithi061.zcjido (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:17.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:16 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:16.776623+0000 mon.smithi061 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi061.zcjido(active, since 2s) 2023-11-16T14:01:17.295 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.290+0000 7f9e4290c700 1 Processor -- start 2023-11-16T14:01:17.295 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e4290c700 1 -- start start 2023-11-16T14:01:17.295 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e4290c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e4290c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c0fcc00 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56954/0 (socket says 172.21.15.61:56954) 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3bfff700 1 -- 172.21.15.61:0/1208709710 learned_addr learned my addr 172.21.15.61:0/1208709710 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3bfff700 1 -- 172.21.15.61:0/1208709710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c0fcd40 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3bfff700 1 --2- 172.21.15.61:0/1208709710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9e2c00b870 tx=0x7f9e2c006f70).ready entity=mon.0 client_cookie=a831ac77099f791d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.291+0000 7f9e3affd700 1 -- 172.21.15.61:0/1208709710 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c00da20 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e3affd700 1 -- 172.21.15.61:0/1208709710 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9e2c010470 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e3affd700 1 -- 172.21.15.61:0/1208709710 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c00f5b0 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 -- 172.21.15.61:0/1208709710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 msgr2=0x7f9e3c0fc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 --2- 172.21.15.61:0/1208709710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9e2c00b870 tx=0x7f9e2c006f70).stop 2023-11-16T14:01:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 -- 172.21.15.61:0/1208709710 shutdown_connections 2023-11-16T14:01:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 --2- 172.21.15.61:0/1208709710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c0fc710 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 -- 172.21.15.61:0/1208709710 >> 172.21.15.61:0/1208709710 conn(0x7f9e3c0f5cb0 msgr2=0x7f9e3c0f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 -- 172.21.15.61:0/1208709710 shutdown_connections 2023-11-16T14:01:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.292+0000 7f9e4290c700 1 -- 172.21.15.61:0/1208709710 wait complete. 2023-11-16T14:01:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.293+0000 7f9e4290c700 1 Processor -- start 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.293+0000 7f9e4290c700 1 -- start start 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.293+0000 7f9e4290c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.293+0000 7f9e4290c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c10d2f0 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.293+0000 7f9e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56962/0 (socket says 172.21.15.61:56962) 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e3bfff700 1 -- 172.21.15.61:0/90078322 learned_addr learned my addr 172.21.15.61:0/90078322 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e3bfff700 1 -- 172.21.15.61:0/90078322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c00b480 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e3bfff700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9e2c00daf0 tx=0x7f9e2c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c00f980 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9e2c019450 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3c10d4f0 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c022620 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.294+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c10e100 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.295+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f9e2c016030 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.295+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3c0591e0 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.295+0000 7f9e38ff9700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 0x7f9e280334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.295+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f9e2c043380 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.295+0000 7f9e3b7fe700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 0x7f9e280334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.296+0000 7f9e3b7fe700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 0x7f9e280334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9e300096a0 tx=0x7f9e30006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:17.302 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.300+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9e2c017360 con 0x7f9e3c0fc2f0 2023-11-16T14:01:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.461+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e3c10e340 con 0x7f9e28030ff0 2023-11-16T14:01:17.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.554+0000 7f9e38ff9700 1 -- 172.21.15.61:0/90078322 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9e3c10e340 con 0x7f9e28030ff0 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 msgr2=0x7f9e280334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 0x7f9e280334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9e300096a0 tx=0x7f9e30006b40).stop 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 msgr2=0x7f9e3c10ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9e2c00daf0 tx=0x7f9e2c008040).stop 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 shutdown_connections 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f9e28030ff0 0x7f9e280334b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 --2- 172.21.15.61:0/90078322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e3c0fc2f0 0x7f9e3c10ce00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.555+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 >> 172.21.15.61:0/90078322 conn(0x7f9e3c0f5cb0 msgr2=0x7f9e3c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:17.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.556+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 shutdown_connections 2023-11-16T14:01:17.560 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:17.556+0000 7f9e4290c700 1 -- 172.21.15.61:0/90078322 wait complete. 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecffd6700 1 Processor -- start 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecffd6700 1 -- start start 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecffd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecffd6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec8072070 con 0x7f5ec8071c50 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecdd72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.224+0000 7f5ecdd72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44238/0 (socket says 172.21.15.61:44238) 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5ecdd72700 1 -- 172.21.15.61:0/3063939513 learned_addr learned my addr 172.21.15.61:0/3063939513 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5ecdd72700 1 -- 172.21.15.61:0/3063939513 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8070640 con 0x7f5ec8071c50 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5ecdd72700 1 --2- 172.21.15.61:0/3063939513 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5eb800b5f0 tx=0x7f5eb8006ee0).ready entity=mon.0 client_cookie=f6def5b874344032 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:18.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5eccd70700 1 -- 172.21.15.61:0/3063939513 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb800d600 con 0x7f5ec8071c50 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5eccd70700 1 -- 172.21.15.61:0/3063939513 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5eb800dbc0 con 0x7f5ec8071c50 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.225+0000 7f5eccd70700 1 -- 172.21.15.61:0/3063939513 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb8011b90 con 0x7f5ec8071c50 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- 172.21.15.61:0/3063939513 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 msgr2=0x7f5ec8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/3063939513 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5eb800b5f0 tx=0x7f5eb8006ee0).stop 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- 172.21.15.61:0/3063939513 shutdown_connections 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/3063939513 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec8070150 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- 172.21.15.61:0/3063939513 >> 172.21.15.61:0/3063939513 conn(0x7f5ec80f7650 msgr2=0x7f5ec80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- 172.21.15.61:0/3063939513 shutdown_connections 2023-11-16T14:01:18.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- 172.21.15.61:0/3063939513 wait complete. 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 Processor -- start 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.226+0000 7f5ecffd6700 1 -- start start 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecffd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecffd6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec810d370 con 0x7f5ec8071c50 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecdd72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecdd72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44252/0 (socket says 172.21.15.61:44252) 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecdd72700 1 -- 172.21.15.61:0/731637879 learned_addr learned my addr 172.21.15.61:0/731637879 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecdd72700 1 -- 172.21.15.61:0/731637879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb800b200 con 0x7f5ec8071c50 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecdd72700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5eb800e040 tx=0x7f5eb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb801b570 con 0x7f5ec8071c50 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5eb801bb30 con 0x7f5ec8071c50 2023-11-16T14:01:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb801a820 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec810d570 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.227+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec810da00 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.228+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f5eb802b070 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.228+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec8045bb0 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.228+0000 7f5eb67fc700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 0x7f5eb0033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.229+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5eb8042ea0 con 0x7f5ec8071c50 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.229+0000 7f5ecd571700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 0x7f5eb0033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:18.234 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.229+0000 7f5ecd571700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 0x7f5eb0033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5ec40096a0 tx=0x7f5ec4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:18.235 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.233+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5eb802b950 con 0x7f5ec8071c50 2023-11-16T14:01:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.394+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ec810e260 con 0x7f5eb0030fa0 2023-11-16T14:01:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.395+0000 7f5eb67fc700 1 -- 172.21.15.61:0/731637879 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5ec810e260 con 0x7f5eb0030fa0 2023-11-16T14:01:18.398 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuAqycKkdidqQUDUnpngVlA1vZPUjRvXU+DFGxfVKi0R4AuVJYW/ytKPIOL8scDKOuaoFVCKAd+6SV9JYSO9SDI2RnkfnRDFwK6JmkdiIxS5uW0f864dm7dVGUjv1N3LRGPbQaw5ZB18z4u/R7DTkr4CmhxgAplebbIaUi8vJTRY0xSTB7xNLHzd1BJGMmGOSuemqxsK5flZSMjYg4jRXxJWAzapxCV/8i4/3v8ViTFAJ9OIYedaFanx+n61zY47OeRKGhcLaA7NUaWUTfH2bijUgaOJWGhjNCttD6+ahTeVWOyGZfqi68SNmf7f4TWq96hLltY2J+TG+9KYKApaaugfzQyndBYTbX+nXwKN4j3yZhpD/XFLEbbUIzD9lO8czfmbiyIT6c2OJwVD232zyULYmXCnIQI5Sd3c2BdRouXBvt9nYkCrRA3VralX3JgtjxOBX/Si6HGzwkexHoGNcBakR+Uk01IGshdG9xpbEafX3B869yYG+Qx6yXaaH8rzE= ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 msgr2=0x7f5eb0033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 0x7f5eb0033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5ec40096a0 tx=0x7f5ec4006b40).stop 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 msgr2=0x7f5ec810ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5eb800e040 tx=0x7f5eb8006ee0).stop 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 shutdown_connections 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f5eb0030fa0 0x7f5eb0033460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 --2- 172.21.15.61:0/731637879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ec8071c50 0x7f5ec810ce80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 >> 172.21.15.61:0/731637879 conn(0x7f5ec80f7650 msgr2=0x7f5ec80f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 shutdown_connections 2023-11-16T14:01:18.401 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:18.396+0000 7f5ecffd6700 1 -- 172.21.15.61:0/731637879 wait complete. 2023-11-16T14:01:18.730 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T14:01:18.731 INFO:teuthology.orchestra.run.smithi061.stderr:Adding key to root@localhost authorized_keys... 2023-11-16T14:01:18.731 INFO:teuthology.orchestra.run.smithi061.stderr:Adding host smithi061... 2023-11-16T14:01:18.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:17.462888+0000 mgr.smithi061.zcjido (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-16T14:01:18.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:17.552621+0000 mon.smithi061 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:18.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:17.554295+0000 mon.smithi061 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.088+0000 7fc0228a7700 1 Processor -- start 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.088+0000 7fc0228a7700 1 -- start start 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.088+0000 7fc0228a7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.088+0000 7fc0228a7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0ff000 con 0x7fc01c06ade0 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:19.093 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44256/0 (socket says 172.21.15.61:44256) 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01bfff700 1 -- 172.21.15.61:0/893914945 learned_addr learned my addr 172.21.15.61:0/893914945 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01bfff700 1 -- 172.21.15.61:0/893914945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0ff140 con 0x7fc01c06ade0 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01bfff700 1 --2- 172.21.15.61:0/893914945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc00c00b5f0 tx=0x7fc00c006ee0).ready entity=mon.0 client_cookie=b8396af1a2558a1d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01affd700 1 -- 172.21.15.61:0/893914945 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c00d600 con 0x7fc01c06ade0 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01affd700 1 -- 172.21.15.61:0/893914945 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc00c00dbc0 con 0x7fc01c06ade0 2023-11-16T14:01:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.089+0000 7fc01affd700 1 -- 172.21.15.61:0/893914945 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c011b90 con 0x7fc01c06ade0 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 -- 172.21.15.61:0/893914945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 msgr2=0x7fc01c0feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 --2- 172.21.15.61:0/893914945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc00c00b5f0 tx=0x7fc00c006ee0).stop 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 -- 172.21.15.61:0/893914945 shutdown_connections 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 --2- 172.21.15.61:0/893914945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c0feb10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 -- 172.21.15.61:0/893914945 >> 172.21.15.61:0/893914945 conn(0x7fc01c0f6590 msgr2=0x7fc01c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 -- 172.21.15.61:0/893914945 shutdown_connections 2023-11-16T14:01:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 -- 172.21.15.61:0/893914945 wait complete. 2023-11-16T14:01:19.096 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.090+0000 7fc0228a7700 1 Processor -- start 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc0228a7700 1 -- start start 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc0228a7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc0228a7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c10b1e0 con 0x7fc01c06ade0 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44258/0 (socket says 172.21.15.61:44258) 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc01bfff700 1 -- 172.21.15.61:0/3217034603 learned_addr learned my addr 172.21.15.61:0/3217034603 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc01bfff700 1 -- 172.21.15.61:0/3217034603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c00b200 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc01bfff700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc00c006800 tx=0x7fc00c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c00de50 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc00c01b870 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c10b3e0 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c01a7a0 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.091+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c10bf90 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.092+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fc00c00f070 con 0x7fc01c06ade0 2023-11-16T14:01:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.093+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c0591e0 con 0x7fc01c06ade0 2023-11-16T14:01:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.093+0000 7fc018ff9700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 0x7fbffc0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.093+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc00c042bc0 con 0x7fc01c06ade0 2023-11-16T14:01:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.093+0000 7fc01b7fe700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 0x7fbffc0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.093+0000 7fc01b7fe700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 0x7fbffc0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc0040096a0 tx=0x7fc004006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.097+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc00c02b8e0 con 0x7fc01c06ade0 2023-11-16T14:01:19.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.261+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}) v1 -- 0x7fc01c10c1d0 con 0x7fbffc030ff0 2023-11-16T14:01:19.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:18.396058+0000 mgr.smithi061.zcjido (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:19.870 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Added host 'smithi061' with addr '172.21.15.61' 2023-11-16T14:01:19.870 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.867+0000 7fc018ff9700 1 -- 172.21.15.61:0/3217034603 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc01c10c1d0 con 0x7fbffc030ff0 2023-11-16T14:01:19.872 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 msgr2=0x7fbffc0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:19.872 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 0x7fbffc0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc0040096a0 tx=0x7fc004006b40).stop 2023-11-16T14:01:19.872 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 msgr2=0x7fc01c10acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc00c006800 tx=0x7fc00c006ee0).stop 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 shutdown_connections 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fbffc030ff0 0x7fbffc0334b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 --2- 172.21.15.61:0/3217034603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc01c06ade0 0x7fc01c10acf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 >> 172.21.15.61:0/3217034603 conn(0x7fc01c0f6590 msgr2=0x7fc01c0f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 shutdown_connections 2023-11-16T14:01:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:19.869+0000 7fc0228a7700 1 -- 172.21.15.61:0/3217034603 wait complete. 2023-11-16T14:01:20.180 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying mon service with default placement... 2023-11-16T14:01:20.531 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.527+0000 7fcb90eab700 1 Processor -- start 2023-11-16T14:01:20.531 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.527+0000 7fcb90eab700 1 -- start start 2023-11-16T14:01:20.531 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb90eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:20.531 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb90eab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0fb710 con 0x7fcb8c0fae00 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb8a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb8a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44274/0 (socket says 172.21.15.61:44274) 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb8a59c700 1 -- 172.21.15.61:0/3231854774 learned_addr learned my addr 172.21.15.61:0/3231854774 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb8a59c700 1 -- 172.21.15.61:0/3231854774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0fb850 con 0x7fcb8c0fae00 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.528+0000 7fcb8a59c700 1 --2- 172.21.15.61:0/3231854774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcb7c00b870 tx=0x7fcb7c006f70).ready entity=mon.0 client_cookie=d42ef459dca01a27 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb8959a700 1 -- 172.21.15.61:0/3231854774 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb7c00da20 con 0x7fcb8c0fae00 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb8959a700 1 -- 172.21.15.61:0/3231854774 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb7c010470 con 0x7fcb8c0fae00 2023-11-16T14:01:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb8959a700 1 -- 172.21.15.61:0/3231854774 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb7c00f5b0 con 0x7fcb8c0fae00 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 -- 172.21.15.61:0/3231854774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 msgr2=0x7fcb8c0fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 --2- 172.21.15.61:0/3231854774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcb7c00b870 tx=0x7fcb7c006f70).stop 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 -- 172.21.15.61:0/3231854774 shutdown_connections 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 --2- 172.21.15.61:0/3231854774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c0fb220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 -- 172.21.15.61:0/3231854774 >> 172.21.15.61:0/3231854774 conn(0x7fcb8c0f6590 msgr2=0x7fcb8c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 -- 172.21.15.61:0/3231854774 shutdown_connections 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.529+0000 7fcb90eab700 1 -- 172.21.15.61:0/3231854774 wait complete. 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb90eab700 1 Processor -- start 2023-11-16T14:01:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb90eab700 1 -- start start 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb90eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb90eab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c108fd0 con 0x7fcb8c0fae00 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb8a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb8a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44282/0 (socket says 172.21.15.61:44282) 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb8a59c700 1 -- 172.21.15.61:0/563058116 learned_addr learned my addr 172.21.15.61:0/563058116 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.530+0000 7fcb8a59c700 1 -- 172.21.15.61:0/563058116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c00b480 con 0x7fcb8c0fae00 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb8a59c700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcb7c006c70 tx=0x7fcb7c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb7c00f3d0 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb7c00fc80 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb7c019b50 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c1091d0 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c109660 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.531+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fcb7c016030 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.532+0000 7fcb82ffd700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 0x7fcb6c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.532+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fcb7c045700 con 0x7fcb8c0fae00 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.532+0000 7fcb89d9b700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 0x7fcb6c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.532+0000 7fcb89d9b700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 0x7fcb6c033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.532+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c045bb0 con 0x7fcb8c0fae00 2023-11-16T14:01:20.539 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.536+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb7c014070 con 0x7fcb8c0fae00 2023-11-16T14:01:20.707 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.704+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb8c102990 con 0x7fcb6c030fa0 2023-11-16T14:01:20.712 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.709+0000 7fcb82ffd700 1 -- 172.21.15.61:0/563058116 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcb8c102990 con 0x7fcb6c030fa0 2023-11-16T14:01:20.712 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-16T14:01:20.714 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 msgr2=0x7fcb6c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:20.714 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 0x7fcb6c033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).stop 2023-11-16T14:01:20.714 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 msgr2=0x7fcb8c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:20.714 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcb7c006c70 tx=0x7fcb7c006f70).stop 2023-11-16T14:01:20.714 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 shutdown_connections 2023-11-16T14:01:20.715 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb6c030fa0 0x7fcb6c033460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:20.715 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 --2- 172.21.15.61:0/563058116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb8c0fae00 0x7fcb8c108ae0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:20.715 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 >> 172.21.15.61:0/563058116 conn(0x7fcb8c0f6590 msgr2=0x7fcb8c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:20.715 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 shutdown_connections 2023-11-16T14:01:20.715 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:20.711+0000 7fcb90eab700 1 -- 172.21.15.61:0/563058116 wait complete. 2023-11-16T14:01:20.921 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:20 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:19.263152+0000 mgr.smithi061.zcjido (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:20.921 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:20 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:19.867518+0000 mon.smithi061 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:20.921 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:20 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:19.868052+0000 mon.smithi061 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:21.088 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying mgr service with default placement... 2023-11-16T14:01:21.450 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.445+0000 7fc567737700 1 Processor -- start 2023-11-16T14:01:21.450 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc567737700 1 -- start start 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc567737700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc567737700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600ff000 con 0x7fc560027670 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc5654d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc5654d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44286/0 (socket says 172.21.15.61:44286) 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc5654d3700 1 -- 172.21.15.61:0/524109998 learned_addr learned my addr 172.21.15.61:0/524109998 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc5654d3700 1 -- 172.21.15.61:0/524109998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600ff140 con 0x7fc560027670 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.446+0000 7fc5654d3700 1 --2- 172.21.15.61:0/524109998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc55c00b5f0 tx=0x7fc55c006ee0).ready entity=mon.0 client_cookie=7f2de0cebd4eb389 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc553fff700 1 -- 172.21.15.61:0/524109998 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c00d600 con 0x7fc560027670 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc553fff700 1 -- 172.21.15.61:0/524109998 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc55c00dbc0 con 0x7fc560027670 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc553fff700 1 -- 172.21.15.61:0/524109998 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c011b90 con 0x7fc560027670 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 -- 172.21.15.61:0/524109998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 msgr2=0x7fc5600feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 --2- 172.21.15.61:0/524109998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc55c00b5f0 tx=0x7fc55c006ee0).stop 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 -- 172.21.15.61:0/524109998 shutdown_connections 2023-11-16T14:01:21.451 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 --2- 172.21.15.61:0/524109998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc5600feb10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:21.452 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 -- 172.21.15.61:0/524109998 >> 172.21.15.61:0/524109998 conn(0x7fc5600f6590 msgr2=0x7fc5600f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:21.452 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 -- 172.21.15.61:0/524109998 shutdown_connections 2023-11-16T14:01:21.452 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.447+0000 7fc567737700 1 -- 172.21.15.61:0/524109998 wait complete. 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.448+0000 7fc567737700 1 Processor -- start 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.448+0000 7fc567737700 1 -- start start 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.448+0000 7fc567737700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc567737700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56010b200 con 0x7fc560027670 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc5654d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc5654d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44290/0 (socket says 172.21.15.61:44290) 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc5654d3700 1 -- 172.21.15.61:0/2322630341 learned_addr learned my addr 172.21.15.61:0/2322630341 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc5654d3700 1 -- 172.21.15.61:0/2322630341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c00b200 con 0x7fc560027670 2023-11-16T14:01:21.454 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc5654d3700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc55c006800 tx=0x7fc55c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c00de50 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc55c01b870 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56010b400 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c01a7a0 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.449+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56010bfb0 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.450+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fc55c00f070 con 0x7fc560027670 2023-11-16T14:01:21.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.450+0000 7fc551ffb700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 0x7fc5480334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.451+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc55c042bb0 con 0x7fc560027670 2023-11-16T14:01:23.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.451+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5600591e0 con 0x7fc560027670 2023-11-16T14:01:23.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.451+0000 7fc564cd2700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 0x7fc5480334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.451+0000 7fc564cd2700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 0x7fc5480334b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5540096a0 tx=0x7fc554006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.455+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc55c016030 con 0x7fc560027670 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.623+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5600fffb0 con 0x7fc548030ff0 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.628+0000 7fc551ffb700 1 -- 172.21.15.61:0/2322630341 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc5600fffb0 con 0x7fc548030ff0 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 msgr2=0x7fc5480334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 0x7fc5480334b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5540096a0 tx=0x7fc554006b40).stop 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 msgr2=0x7fc56010ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc55c006800 tx=0x7fc55c006ee0).stop 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 shutdown_connections 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc548030ff0 0x7fc5480334b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 --2- 172.21.15.61:0/2322630341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc560027670 0x7fc56010ad10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 >> 172.21.15.61:0/2322630341 conn(0x7fc5600f6590 msgr2=0x7fc5600f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 shutdown_connections 2023-11-16T14:01:23.933 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:21.630+0000 7fc567737700 1 -- 172.21.15.61:0/2322630341 wait complete. 2023-11-16T14:01:23.933 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:19.867793+0000 mgr.smithi061.zcjido (mgr.14116) 9 : cephadm [INF] Added host smithi061 2023-11-16T14:01:23.933 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:20.710083+0000 mon.smithi061 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying crash service with default placement... 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.293+0000 7f4af9131700 1 Processor -- start 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.293+0000 7f4af9131700 1 -- start start 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.293+0000 7f4af9131700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.293+0000 7f4af9131700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af40fb710 con 0x7f4af40fae00 2023-11-16T14:01:23.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.293+0000 7f4af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44296/0 (socket says 172.21.15.61:44296) 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af2d9d700 1 -- 172.21.15.61:0/535505678 learned_addr learned my addr 172.21.15.61:0/535505678 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af2d9d700 1 -- 172.21.15.61:0/535505678 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40fb850 con 0x7f4af40fae00 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af2d9d700 1 --2- 172.21.15.61:0/535505678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4ae800b5f0 tx=0x7f4ae8006ee0).ready entity=mon.0 client_cookie=e89d5eebc513b350 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af1d9b700 1 -- 172.21.15.61:0/535505678 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae800d600 con 0x7f4af40fae00 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af1d9b700 1 -- 172.21.15.61:0/535505678 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4ae800dbc0 con 0x7f4af40fae00 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.294+0000 7f4af1d9b700 1 -- 172.21.15.61:0/535505678 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8011b90 con 0x7f4af40fae00 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 -- 172.21.15.61:0/535505678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 msgr2=0x7f4af40fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 --2- 172.21.15.61:0/535505678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4ae800b5f0 tx=0x7f4ae8006ee0).stop 2023-11-16T14:01:23.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 -- 172.21.15.61:0/535505678 shutdown_connections 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 --2- 172.21.15.61:0/535505678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af40fb220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 -- 172.21.15.61:0/535505678 >> 172.21.15.61:0/535505678 conn(0x7f4af40f6590 msgr2=0x7f4af40f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 -- 172.21.15.61:0/535505678 shutdown_connections 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.295+0000 7f4af9131700 1 -- 172.21.15.61:0/535505678 wait complete. 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af9131700 1 Processor -- start 2023-11-16T14:01:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af9131700 1 -- start start 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af9131700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af9131700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af410b1f0 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44298/0 (socket says 172.21.15.61:44298) 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af2d9d700 1 -- 172.21.15.61:0/1683595414 learned_addr learned my addr 172.21.15.61:0/1683595414 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af2d9d700 1 -- 172.21.15.61:0/1683595414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae800b200 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.296+0000 7f4af2d9d700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4ae8000c00 tx=0x7f4ae8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.297+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae800de00 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.297+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4ae801b870 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.297+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af410b3f0 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.297+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae801a670 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.297+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af410bfa0 con 0x7f4af40fae00 2023-11-16T14:01:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f4ae802b070 con 0x7f4af40fae00 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af40591e0 con 0x7f4af40fae00 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4adb7fe700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 0x7f4ae00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f4ae8042cb0 con 0x7f4af40fae00 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4af259c700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 0x7f4ae00334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.298+0000 7f4af259c700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 0x7f4ae00334b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.302+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4ae802b8f0 con 0x7f4af40fae00 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.470+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af410c2a0 con 0x7f4ae0030ff0 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.473+0000 7f4adb7fe700 1 -- 172.21.15.61:0/1683595414 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4af410c2a0 con 0x7f4ae0030ff0 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 msgr2=0x7f4ae00334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 0x7f4ae00334b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).stop 2023-11-16T14:01:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 msgr2=0x7f4af410ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4ae8000c00 tx=0x7f4ae8006ee0).stop 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 shutdown_connections 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f4ae0030ff0 0x7f4ae00334b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 --2- 172.21.15.61:0/1683595414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4af40fae00 0x7f4af410ad00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 >> 172.21.15.61:0/1683595414 conn(0x7f4af40f6590 msgr2=0x7f4af40271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 shutdown_connections 2023-11-16T14:01:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:22.475+0000 7f4af9131700 1 -- 172.21.15.61:0/1683595414 wait complete. 2023-11-16T14:01:23.963 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:20.706359+0000 mgr.smithi061.zcjido (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:23.963 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:20.707121+0000 mgr.smithi061.zcjido (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-16T14:01:23.963 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:21.629254+0000 mon.smithi061 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:23.964 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:22.474609+0000 mon.smithi061 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:23.968 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying prometheus service with default placement... 2023-11-16T14:01:23.968 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc376007700 1 Processor -- start 2023-11-16T14:01:23.968 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc376007700 1 -- start start 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc376007700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc376007700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700fee70 con 0x7fc37006ade0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc375005700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc375005700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44312/0 (socket says 172.21.15.61:44312) 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.186+0000 7fc375005700 1 -- 172.21.15.61:0/3940893530 learned_addr learned my addr 172.21.15.61:0/3940893530 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.187+0000 7fc375005700 1 -- 172.21.15.61:0/3940893530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700fefb0 con 0x7fc37006ade0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.187+0000 7fc375005700 1 --2- 172.21.15.61:0/3940893530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc36c00b5f0 tx=0x7fc36c006ee0).ready entity=mon.0 client_cookie=e60708e01798372 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.187+0000 7fc35ffff700 1 -- 172.21.15.61:0/3940893530 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c00d600 con 0x7fc37006ade0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.187+0000 7fc35ffff700 1 -- 172.21.15.61:0/3940893530 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc36c00dbc0 con 0x7fc37006ade0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.187+0000 7fc35ffff700 1 -- 172.21.15.61:0/3940893530 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c011b90 con 0x7fc37006ade0 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- 172.21.15.61:0/3940893530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 msgr2=0x7fc3700fe980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 --2- 172.21.15.61:0/3940893530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc36c00b5f0 tx=0x7fc36c006ee0).stop 2023-11-16T14:01:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- 172.21.15.61:0/3940893530 shutdown_connections 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 --2- 172.21.15.61:0/3940893530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc3700fe980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- 172.21.15.61:0/3940893530 >> 172.21.15.61:0/3940893530 conn(0x7fc3700f63c0 msgr2=0x7fc3700f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- 172.21.15.61:0/3940893530 shutdown_connections 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- 172.21.15.61:0/3940893530 wait complete. 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 Processor -- start 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.188+0000 7fc376007700 1 -- start start 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc376007700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc376007700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37010afb0 con 0x7fc37006ade0 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc375005700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc375005700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44324/0 (socket says 172.21.15.61:44324) 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc375005700 1 -- 172.21.15.61:0/344270201 learned_addr learned my addr 172.21.15.61:0/344270201 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc375005700 1 -- 172.21.15.61:0/344270201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c00b200 con 0x7fc37006ade0 2023-11-16T14:01:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc375005700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc36c000c00 tx=0x7fc36c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c01b410 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc36c01b9d0 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c01a620 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.189+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37010b1b0 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.190+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37010b640 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.190+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fc36c02b070 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.190+0000 7fc35dffb700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 0x7fc358033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.190+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370045bb0 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.190+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc36c042c10 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.191+0000 7fc374804700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 0x7fc358033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.191+0000 7fc374804700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 0x7fc358033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc3600096a0 tx=0x7fc360006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.195+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc36c016020 con 0x7fc37006ade0 2023-11-16T14:01:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.366+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37010bf60 con 0x7fc358030fa0 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.370+0000 7fc35dffb700 1 -- 172.21.15.61:0/344270201 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc37010bf60 con 0x7fc358030fa0 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 msgr2=0x7fc358033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 0x7fc358033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc3600096a0 tx=0x7fc360006b40).stop 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 msgr2=0x7fc37010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc36c000c00 tx=0x7fc36c006ee0).stop 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 shutdown_connections 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fc358030fa0 0x7fc358033460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 --2- 172.21.15.61:0/344270201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37006ade0 0x7fc37010aac0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 >> 172.21.15.61:0/344270201 conn(0x7fc3700f63c0 msgr2=0x7fc3700f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 shutdown_connections 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:23.372+0000 7fc376007700 1 -- 172.21.15.61:0/344270201 wait complete. 2023-11-16T14:01:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying grafana service with default placement... 2023-11-16T14:01:23.973 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:21.625587+0000 mgr.smithi061.zcjido (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:23.973 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:21.626094+0000 mgr.smithi061.zcjido (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-16T14:01:23.973 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:22.471769+0000 mgr.smithi061.zcjido (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:23.973 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:22.472442+0000 mgr.smithi061.zcjido (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-16T14:01:23.973 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:23.371461+0000 mon.smithi061 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:24.036 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.032+0000 7fa13359e700 1 Processor -- start 2023-11-16T14:01:24.036 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13359e700 1 -- start start 2023-11-16T14:01:24.036 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1340fc810 con 0x7fa13406ad40 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44334/0 (socket says 172.21.15.61:44334) 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13259c700 1 -- 172.21.15.61:0/4276358413 learned_addr learned my addr 172.21.15.61:0/4276358413 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.033+0000 7fa13259c700 1 -- 172.21.15.61:0/4276358413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa134101310 con 0x7fa13406ad40 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13259c700 1 --2- 172.21.15.61:0/4276358413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa12400b5f0 tx=0x7fa124006ee0).ready entity=mon.0 client_cookie=e666c6883150b9c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13159a700 1 -- 172.21.15.61:0/4276358413 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa12400d600 con 0x7fa13406ad40 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13159a700 1 -- 172.21.15.61:0/4276358413 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa12400dbc0 con 0x7fa13406ad40 2023-11-16T14:01:24.037 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13159a700 1 -- 172.21.15.61:0/4276358413 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa124011b90 con 0x7fa13406ad40 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 -- 172.21.15.61:0/4276358413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 msgr2=0x7fa134100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 --2- 172.21.15.61:0/4276358413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa12400b5f0 tx=0x7fa124006ee0).stop 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 -- 172.21.15.61:0/4276358413 shutdown_connections 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 --2- 172.21.15.61:0/4276358413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa134100e20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 -- 172.21.15.61:0/4276358413 >> 172.21.15.61:0/4276358413 conn(0x7fa1340f63c0 msgr2=0x7fa1340f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 -- 172.21.15.61:0/4276358413 shutdown_connections 2023-11-16T14:01:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.034+0000 7fa13359e700 1 -- 172.21.15.61:0/4276358413 wait complete. 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.035+0000 7fa13359e700 1 Processor -- start 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.035+0000 7fa13359e700 1 -- start start 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.035+0000 7fa13359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.035+0000 7fa13359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13410d260 con 0x7fa13406ad40 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44338/0 (socket says 172.21.15.61:44338) 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13259c700 1 -- 172.21.15.61:0/305428310 learned_addr learned my addr 172.21.15.61:0/305428310 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13259c700 1 -- 172.21.15.61:0/305428310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12400b200 con 0x7fa13406ad40 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13259c700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa124006800 tx=0x7fa124006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa12400d9c0 con 0x7fa13406ad40 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa12401b870 con 0x7fa13406ad40 2023-11-16T14:01:24.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13410d460 con 0x7fa13406ad40 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa12401a740 con 0x7fa13406ad40 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.036+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13410e070 con 0x7fa13406ad40 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.037+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa12402b070 con 0x7fa13406ad40 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.037+0000 7fa12affd700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 0x7fa1140334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.037+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa124042bf0 con 0x7fa13406ad40 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.037+0000 7fa131d9b700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 0x7fa1140334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.038+0000 7fa131d9b700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 0x7fa1140334b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa11c0096a0 tx=0x7fa11c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.042 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.038+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1340591e0 con 0x7fa13406ad40 2023-11-16T14:01:24.044 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.042+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa124016020 con 0x7fa13406ad40 2023-11-16T14:01:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.210+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1340008d0 con 0x7fa114030ff0 2023-11-16T14:01:24.217 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.214+0000 7fa12affd700 1 -- 172.21.15.61:0/305428310 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa1340008d0 con 0x7fa114030ff0 2023-11-16T14:01:24.217 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-16T14:01:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 msgr2=0x7fa1140334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 0x7fa1140334b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa11c0096a0 tx=0x7fa11c006b40).stop 2023-11-16T14:01:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 msgr2=0x7fa13410cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa124006800 tx=0x7fa124006ee0).stop 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 shutdown_connections 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa114030ff0 0x7fa1140334b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 --2- 172.21.15.61:0/305428310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa13406ad40 0x7fa13410cd70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.216+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 >> 172.21.15.61:0/305428310 conn(0x7fa1340f63c0 msgr2=0x7fa1340f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.217+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 shutdown_connections 2023-11-16T14:01:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.217+0000 7fa13359e700 1 -- 172.21.15.61:0/305428310 wait complete. 2023-11-16T14:01:24.554 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying node-exporter service with default placement... 2023-11-16T14:01:24.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:23.368640+0000 mgr.smithi061.zcjido (mgr.14116) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:24.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:23.369194+0000 mgr.smithi061.zcjido (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-16T14:01:24.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:24.215212+0000 mon.smithi061 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.920+0000 7fcb711dd700 1 Processor -- start 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.920+0000 7fcb711dd700 1 -- start start 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.920+0000 7fcb711dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb711dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ff030 con 0x7fcb6c070c80 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44346/0 (socket says 172.21.15.61:44346) 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb6ad9d700 1 -- 172.21.15.61:0/4034997801 learned_addr learned my addr 172.21.15.61:0/4034997801 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb6ad9d700 1 -- 172.21.15.61:0/4034997801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0ff170 con 0x7fcb6c070c80 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb6ad9d700 1 --2- 172.21.15.61:0/4034997801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcb5c00b5f0 tx=0x7fcb5c006ee0).ready entity=mon.0 client_cookie=b18b633ba614e501 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb69d9b700 1 -- 172.21.15.61:0/4034997801 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c00d600 con 0x7fcb6c070c80 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb69d9b700 1 -- 172.21.15.61:0/4034997801 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb5c00dbc0 con 0x7fcb6c070c80 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.921+0000 7fcb69d9b700 1 -- 172.21.15.61:0/4034997801 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c011b90 con 0x7fcb6c070c80 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 -- 172.21.15.61:0/4034997801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 msgr2=0x7fcb6c0feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 --2- 172.21.15.61:0/4034997801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcb5c00b5f0 tx=0x7fcb5c006ee0).stop 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 -- 172.21.15.61:0/4034997801 shutdown_connections 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 --2- 172.21.15.61:0/4034997801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c0feb40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 -- 172.21.15.61:0/4034997801 >> 172.21.15.61:0/4034997801 conn(0x7fcb6c0f6560 msgr2=0x7fcb6c0f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 -- 172.21.15.61:0/4034997801 shutdown_connections 2023-11-16T14:01:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.922+0000 7fcb711dd700 1 -- 172.21.15.61:0/4034997801 wait complete. 2023-11-16T14:01:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb711dd700 1 Processor -- start 2023-11-16T14:01:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb711dd700 1 -- start start 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb711dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb711dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c10b1e0 con 0x7fcb6c070c80 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44350/0 (socket says 172.21.15.61:44350) 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.923+0000 7fcb6ad9d700 1 -- 172.21.15.61:0/1985997914 learned_addr learned my addr 172.21.15.61:0/1985997914 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb6ad9d700 1 -- 172.21.15.61:0/1985997914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c00b200 con 0x7fcb6c070c80 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb6ad9d700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcb5c006800 tx=0x7fcb5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c00de50 con 0x7fcb6c070c80 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb5c01b870 con 0x7fcb6c070c80 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c10b3e0 con 0x7fcb6c070c80 2023-11-16T14:01:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c01a7a0 con 0x7fcb6c070c80 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.924+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c10bf90 con 0x7fcb6c070c80 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.925+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fcb5c00f070 con 0x7fcb6c070c80 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.925+0000 7fcb637fe700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 0x7fcb4c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.925+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fcb5c042bf0 con 0x7fcb6c070c80 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.925+0000 7fcb6a59c700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 0x7fcb4c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.926+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c0591e0 con 0x7fcb6c070c80 2023-11-16T14:01:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.926+0000 7fcb6a59c700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 0x7fcb4c033460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb540096a0 tx=0x7fcb54006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:24.932 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:24.930+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb5c016030 con 0x7fcb6c070c80 2023-11-16T14:01:25.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.098+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb6c0ffef0 con 0x7fcb4c030fa0 2023-11-16T14:01:25.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.102+0000 7fcb637fe700 1 -- 172.21.15.61:0/1985997914 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fcb6c0ffef0 con 0x7fcb4c030fa0 2023-11-16T14:01:25.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-16T14:01:25.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 msgr2=0x7fcb4c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:25.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 0x7fcb4c033460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb540096a0 tx=0x7fcb54006b40).stop 2023-11-16T14:01:25.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 msgr2=0x7fcb6c10acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:25.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcb5c006800 tx=0x7fcb5c006ee0).stop 2023-11-16T14:01:25.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 shutdown_connections 2023-11-16T14:01:25.108 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcb4c030fa0 0x7fcb4c033460 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:25.108 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 --2- 172.21.15.61:0/1985997914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb6c070c80 0x7fcb6c10acf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:25.108 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 >> 172.21.15.61:0/1985997914 conn(0x7fcb6c0f6560 msgr2=0x7fcb6c0f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:25.108 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 shutdown_connections 2023-11-16T14:01:25.108 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.104+0000 7fcb711dd700 1 -- 172.21.15.61:0/1985997914 wait complete. 2023-11-16T14:01:25.438 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying alertmanager service with default placement... 2023-11-16T14:01:25.772 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.765+0000 7fa2857e6700 1 Processor -- start 2023-11-16T14:01:25.772 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.765+0000 7fa2857e6700 1 -- start start 2023-11-16T14:01:25.772 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.765+0000 7fa2857e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.765+0000 7fa2857e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa280072070 con 0x7fa280071c50 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.765+0000 7fa27effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44356/0 (socket says 172.21.15.61:44356) 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27effd700 1 -- 172.21.15.61:0/339410056 learned_addr learned my addr 172.21.15.61:0/339410056 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27effd700 1 -- 172.21.15.61:0/339410056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280070640 con 0x7fa280071c50 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27effd700 1 --2- 172.21.15.61:0/339410056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa27400b5f0 tx=0x7fa274006ee0).ready entity=mon.0 client_cookie=ef114ddbf5dbdcf4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27dffb700 1 -- 172.21.15.61:0/339410056 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27400d600 con 0x7fa280071c50 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27dffb700 1 -- 172.21.15.61:0/339410056 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa27400dbc0 con 0x7fa280071c50 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.766+0000 7fa27dffb700 1 -- 172.21.15.61:0/339410056 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa274011b90 con 0x7fa280071c50 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- 172.21.15.61:0/339410056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 msgr2=0x7fa280070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 --2- 172.21.15.61:0/339410056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa27400b5f0 tx=0x7fa274006ee0).stop 2023-11-16T14:01:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- 172.21.15.61:0/339410056 shutdown_connections 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 --2- 172.21.15.61:0/339410056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa280070150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- 172.21.15.61:0/339410056 >> 172.21.15.61:0/339410056 conn(0x7fa2800f7500 msgr2=0x7fa2800f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- 172.21.15.61:0/339410056 shutdown_connections 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- 172.21.15.61:0/339410056 wait complete. 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 Processor -- start 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.767+0000 7fa2857e6700 1 -- start start 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2857e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2857e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28010d230 con 0x7fa280071c50 2023-11-16T14:01:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa27effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa27effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44360/0 (socket says 172.21.15.61:44360) 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa27effd700 1 -- 172.21.15.61:0/4122452151 learned_addr learned my addr 172.21.15.61:0/4122452151 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa27effd700 1 -- 172.21.15.61:0/4122452151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27400b200 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa27effd700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa27400e040 tx=0x7fa274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27401b570 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa27401bb30 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28010d430 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27401a820 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.768+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28010e040 con 0x7fa280071c50 2023-11-16T14:01:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.769+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa27402b070 con 0x7fa280071c50 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.769+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2800591e0 con 0x7fa280071c50 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.769+0000 7fa2677fe700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 0x7fa26c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.769+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa274042e60 con 0x7fa280071c50 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.769+0000 7fa27e7fc700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 0x7fa26c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.770+0000 7fa27e7fc700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 0x7fa26c033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa2700096a0 tx=0x7fa270006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.774+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa27401b6d0 con 0x7fa280071c50 2023-11-16T14:01:25.946 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.943+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fa280102090 con 0x7fa26c030fa0 2023-11-16T14:01:25.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.947+0000 7fa2677fe700 1 -- 172.21.15.61:0/4122452151 <== mgr.14116 v2:172.21.15.61:6800/1639220556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fa280102090 con 0x7fa26c030fa0 2023-11-16T14:01:25.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-16T14:01:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 msgr2=0x7fa26c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 0x7fa26c033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa2700096a0 tx=0x7fa270006b40).stop 2023-11-16T14:01:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 msgr2=0x7fa28010cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa27400e040 tx=0x7fa274006ee0).stop 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 shutdown_connections 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fa26c030fa0 0x7fa26c033460 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 --2- 172.21.15.61:0/4122452151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa280071c50 0x7fa28010cd40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.949+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 >> 172.21.15.61:0/4122452151 conn(0x7fa2800f7500 msgr2=0x7fa2800f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.950+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 shutdown_connections 2023-11-16T14:01:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:25.950+0000 7fa2857e6700 1 -- 172.21.15.61:0/4122452151 wait complete. 2023-11-16T14:01:25.978 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:24.212546+0000 mgr.smithi061.zcjido (mgr.14116) 18 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:25.978 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:24.213067+0000 mgr.smithi061.zcjido (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-16T14:01:25.979 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:25.103373+0000 mon.smithi061 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea52339700 1 Processor -- start 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea52339700 1 -- start start 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea52339700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea52339700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0ff000 con 0x7fea4c027670 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44374/0 (socket says 172.21.15.61:44374) 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.636+0000 7fea4bfff700 1 -- 172.21.15.61:0/1781915805 learned_addr learned my addr 172.21.15.61:0/1781915805 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.637+0000 7fea4bfff700 1 -- 172.21.15.61:0/1781915805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c0ff140 con 0x7fea4c027670 2023-11-16T14:01:26.641 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.637+0000 7fea4bfff700 1 --2- 172.21.15.61:0/1781915805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fea3c00b5f0 tx=0x7fea3c006ee0).ready entity=mon.0 client_cookie=2820eac571a79a0d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.637+0000 7fea4affd700 1 -- 172.21.15.61:0/1781915805 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea3c00d600 con 0x7fea4c027670 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.637+0000 7fea4affd700 1 -- 172.21.15.61:0/1781915805 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fea3c00dbc0 con 0x7fea4c027670 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.637+0000 7fea4affd700 1 -- 172.21.15.61:0/1781915805 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea3c011b90 con 0x7fea4c027670 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 -- 172.21.15.61:0/1781915805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 msgr2=0x7fea4c0feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 --2- 172.21.15.61:0/1781915805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fea3c00b5f0 tx=0x7fea3c006ee0).stop 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 -- 172.21.15.61:0/1781915805 shutdown_connections 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 --2- 172.21.15.61:0/1781915805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c0feb10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 -- 172.21.15.61:0/1781915805 >> 172.21.15.61:0/1781915805 conn(0x7fea4c0f6590 msgr2=0x7fea4c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 -- 172.21.15.61:0/1781915805 shutdown_connections 2023-11-16T14:01:26.642 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 -- 172.21.15.61:0/1781915805 wait complete. 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.638+0000 7fea52339700 1 Processor -- start 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea52339700 1 -- start start 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea52339700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea52339700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c10b1e0 con 0x7fea4c027670 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:26.645 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44380/0 (socket says 172.21.15.61:44380) 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea4bfff700 1 -- 172.21.15.61:0/3066209509 learned_addr learned my addr 172.21.15.61:0/3066209509 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.639+0000 7fea4bfff700 1 -- 172.21.15.61:0/3066209509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c00b200 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea4bfff700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fea3c000c00 tx=0x7fea3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea3c01b410 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fea3c01b9d0 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c10b3e0 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea3c01a620 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.640+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c10bf90 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.641+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fea3c02b070 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.641+0000 7fea48ff9700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 0x7fea38033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.641+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fea3c043a00 con 0x7fea4c027670 2023-11-16T14:01:26.646 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.641+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c004fa0 con 0x7fea4c027670 2023-11-16T14:01:26.647 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.641+0000 7fea4b7fe700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 0x7fea38033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:26.647 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.642+0000 7fea4b7fe700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 0x7fea38033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fea400096a0 tx=0x7fea40006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:26.648 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.645+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fea3c01f080 con 0x7fea4c027670 2023-11-16T14:01:26.811 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.808+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fea2c005670 con 0x7fea4c027670 2023-11-16T14:01:26.816 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.814+0000 7fea48ff9700 1 -- 172.21.15.61:0/3066209509 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fea3c016020 con 0x7fea4c027670 2023-11-16T14:01:26.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 msgr2=0x7fea38033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:26.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 0x7fea38033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fea400096a0 tx=0x7fea40006b40).stop 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 msgr2=0x7fea4c10acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fea3c000c00 tx=0x7fea3c006ee0).stop 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 shutdown_connections 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fea38030fa0 0x7fea38033460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 --2- 172.21.15.61:0/3066209509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea4c027670 0x7fea4c10acf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 >> 172.21.15.61:0/3066209509 conn(0x7fea4c0f6590 msgr2=0x7fea4c0f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:26.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 shutdown_connections 2023-11-16T14:01:26.821 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:26.816+0000 7fea52339700 1 -- 172.21.15.61:0/3066209509 wait complete. 2023-11-16T14:01:26.842 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:25.100594+0000 mgr.smithi061.zcjido (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:26.843 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:25.101139+0000 mgr.smithi061.zcjido (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-16T14:01:26.843 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:25.948396+0000 mon.smithi061 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.674+0000 7fb0634b8700 1 Processor -- start 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.674+0000 7fb0634b8700 1 -- start start 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.674+0000 7fb0634b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.674+0000 7fb0634b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c0fb710 con 0x7fb05c0fae00 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb061254700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb061254700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49626/0 (socket says 172.21.15.61:49626) 2023-11-16T14:01:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb061254700 1 -- 172.21.15.61:0/3785556606 learned_addr learned my addr 172.21.15.61:0/3785556606 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:27.679 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb061254700 1 -- 172.21.15.61:0/3785556606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c0fb850 con 0x7fb05c0fae00 2023-11-16T14:01:27.679 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb061254700 1 --2- 172.21.15.61:0/3785556606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb04c00b5f0 tx=0x7fb04c006ee0).ready entity=mon.0 client_cookie=a0ea28cb56925ded server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:27.679 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb04bfff700 1 -- 172.21.15.61:0/3785556606 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c00d600 con 0x7fb05c0fae00 2023-11-16T14:01:27.679 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb04bfff700 1 -- 172.21.15.61:0/3785556606 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb04c00dbc0 con 0x7fb05c0fae00 2023-11-16T14:01:27.679 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.675+0000 7fb04bfff700 1 -- 172.21.15.61:0/3785556606 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c011b90 con 0x7fb05c0fae00 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- 172.21.15.61:0/3785556606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 msgr2=0x7fb05c0fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 --2- 172.21.15.61:0/3785556606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb04c00b5f0 tx=0x7fb04c006ee0).stop 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- 172.21.15.61:0/3785556606 shutdown_connections 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 --2- 172.21.15.61:0/3785556606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c0fb220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- 172.21.15.61:0/3785556606 >> 172.21.15.61:0/3785556606 conn(0x7fb05c0f6590 msgr2=0x7fb05c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- 172.21.15.61:0/3785556606 shutdown_connections 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- 172.21.15.61:0/3785556606 wait complete. 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 Processor -- start 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.676+0000 7fb0634b8700 1 -- start start 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb0634b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb0634b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c10d380 con 0x7fb05c0fae00 2023-11-16T14:01:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb061254700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb061254700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49630/0 (socket says 172.21.15.61:49630) 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb061254700 1 -- 172.21.15.61:0/1401250080 learned_addr learned my addr 172.21.15.61:0/1401250080 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb061254700 1 -- 172.21.15.61:0/1401250080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c00b200 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb061254700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb04c006800 tx=0x7fb04c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c00d9c0 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c10d580 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb04c01b870 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.677+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c01a740 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.678+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c10da10 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.678+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fb04c02b070 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.679+0000 7fb049ffb700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 0x7fb044033440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.679+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb04c042c30 con 0x7fb05c0fae00 2023-11-16T14:01:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.679+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c045bb0 con 0x7fb05c0fae00 2023-11-16T14:01:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.679+0000 7fb060a53700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 0x7fb044033440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.679+0000 7fb060a53700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 0x7fb044033440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb0580096a0 tx=0x7fb058006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.683+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb04c016080 con 0x7fb05c0fae00 2023-11-16T14:01:27.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.846+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb05c0fcae0 con 0x7fb05c0fae00 2023-11-16T14:01:27.854 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.851+0000 7fb049ffb700 1 -- 172.21.15.61:0/1401250080 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb04c01f070 con 0x7fb05c0fae00 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 msgr2=0x7fb044033440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 0x7fb044033440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb0580096a0 tx=0x7fb058006b40).stop 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 msgr2=0x7fb05c10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb04c006800 tx=0x7fb04c006ee0).stop 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 shutdown_connections 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fb044030f80 0x7fb044033440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 --2- 172.21.15.61:0/1401250080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb05c0fae00 0x7fb05c10ce90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 >> 172.21.15.61:0/1401250080 conn(0x7fb05c0f6590 msgr2=0x7fb05c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.853+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 shutdown_connections 2023-11-16T14:01:27.857 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:27.854+0000 7fb0634b8700 1 -- 172.21.15.61:0/1401250080 wait complete. 2023-11-16T14:01:27.858 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:25.945727+0000 mgr.smithi061.zcjido (mgr.14116) 22 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:27.858 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:25.946273+0000 mgr.smithi061.zcjido (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-16T14:01:27.858 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:26.815199+0000 mon.smithi061 (mon.0) 64 : audit [INF] from='client.? 172.21.15.61:0/3066209509' entity='client.admin' 2023-11-16T14:01:28.195 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling the dashboard module... 2023-11-16T14:01:28.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.572+0000 7f0aa8038700 1 Processor -- start 2023-11-16T14:01:28.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.572+0000 7f0aa8038700 1 -- start start 2023-11-16T14:01:28.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.572+0000 7f0aa8038700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:28.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.572+0000 7f0aa8038700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00fc7b0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.572+0000 7f0aa5dd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa5dd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49642/0 (socket says 172.21.15.61:49642) 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa5dd4700 1 -- 172.21.15.61:0/1043933363 learned_addr learned my addr 172.21.15.61:0/1043933363 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa5dd4700 1 -- 172.21.15.61:0/1043933363 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00fc8f0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa5dd4700 1 --2- 172.21.15.61:0/1043933363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0a9c00b5f0 tx=0x7f0a9c006ee0).ready entity=mon.0 client_cookie=9f7c66a2427d419c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa4dd2700 1 -- 172.21.15.61:0/1043933363 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9c00d600 con 0x7f0aa00fbea0 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa4dd2700 1 -- 172.21.15.61:0/1043933363 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0a9c00dbf0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.577 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.573+0000 7f0aa4dd2700 1 -- 172.21.15.61:0/1043933363 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9c011bf0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 -- 172.21.15.61:0/1043933363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 msgr2=0x7f0aa00fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 --2- 172.21.15.61:0/1043933363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0a9c00b5f0 tx=0x7f0a9c006ee0).stop 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 -- 172.21.15.61:0/1043933363 shutdown_connections 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 --2- 172.21.15.61:0/1043933363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa00fc2c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 -- 172.21.15.61:0/1043933363 >> 172.21.15.61:0/1043933363 conn(0x7f0aa00f7660 msgr2=0x7f0aa00f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 -- 172.21.15.61:0/1043933363 shutdown_connections 2023-11-16T14:01:28.579 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.574+0000 7f0aa8038700 1 -- 172.21.15.61:0/1043933363 wait complete. 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.575+0000 7f0aa8038700 1 Processor -- start 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.575+0000 7f0aa8038700 1 -- start start 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.575+0000 7f0aa8038700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.575+0000 7f0aa8038700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00711e0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.575+0000 7f0aa5dd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa5dd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49652/0 (socket says 172.21.15.61:49652) 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa5dd4700 1 -- 172.21.15.61:0/3546143642 learned_addr learned my addr 172.21.15.61:0/3546143642 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa5dd4700 1 -- 172.21.15.61:0/3546143642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c00b200 con 0x7f0aa00fbea0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa5dd4700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0a9c011e30 tx=0x7f0a9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9c01b410 con 0x7f0aa00fbea0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0a9c01ba00 con 0x7f0aa00fbea0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa0071380 con 0x7f0aa00fbea0 2023-11-16T14:01:28.580 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9c01a750 con 0x7f0aa00fbea0 2023-11-16T14:01:28.581 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.576+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa0071810 con 0x7f0aa00fbea0 2023-11-16T14:01:28.582 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.577+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0a9c02b070 con 0x7f0aa00fbea0 2023-11-16T14:01:28.582 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.577+0000 7f0a8e7fc700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 0x7f0a880334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:28.582 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.578+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0a9c042b30 con 0x7f0aa00fbea0 2023-11-16T14:01:28.583 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.578+0000 7f0aa55d3700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 0x7f0a880334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:28.583 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.578+0000 7f0aa55d3700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 0x7f0a880334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0a900096a0 tx=0x7f0a90006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:28.584 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.581+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa00591e0 con 0x7f0aa00fbea0 2023-11-16T14:01:28.590 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.587+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0a9c016020 con 0x7f0aa00fbea0 2023-11-16T14:01:28.807 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:28.804+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0aa0045bb0 con 0x7f0aa00fbea0 2023-11-16T14:01:29.109 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:27.852633+0000 mon.smithi061 (mon.0) 65 : audit [INF] from='client.? 172.21.15.61:0/1401250080' entity='client.admin' 2023-11-16T14:01:29.109 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:28.163050+0000 mon.smithi061 (mon.0) 66 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:29.109 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:28.406310+0000 mon.smithi061 (mon.0) 67 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:29.109 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:28.806225+0000 mon.smithi061 (mon.0) 68 : audit [INF] from='client.? 172.21.15.61:0/3546143642' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T14:01:29.109 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:28.833107+0000 mon.smithi061 (mon.0) 69 : audit [INF] from='mgr.14116 172.21.15.61:0/1327458703' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.405+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f0a9c00f070 con 0x7f0aa00fbea0 2023-11-16T14:01:29.409 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.406+0000 7f0a8e7fc700 1 -- 172.21.15.61:0/3546143642 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0a9c022310 con 0x7f0aa00fbea0 2023-11-16T14:01:29.411 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 msgr2=0x7f0a880334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 0x7f0a880334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0a900096a0 tx=0x7f0a90006b40).stop 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 msgr2=0x7f0aa0072d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0a9c011e30 tx=0x7f0a9c006ee0).stop 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 shutdown_connections 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f0a88030ff0 0x7f0a880334b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 --2- 172.21.15.61:0/3546143642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa00fbea0 0x7f0aa0072d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.408+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 >> 172.21.15.61:0/3546143642 conn(0x7f0aa00f7660 msgr2=0x7f0aa00f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.409+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 shutdown_connections 2023-11-16T14:01:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:29.409+0000 7f0aa8038700 1 -- 172.21.15.61:0/3546143642 wait complete. 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.246+0000 7f2696e22700 1 Processor -- start 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.246+0000 7f2696e22700 1 -- start start 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.246+0000 7f2696e22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.246+0000 7f2696e22700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26900ff000 con 0x7f2690027670 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f2694bbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f2694bbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49686/0 (socket says 172.21.15.61:49686) 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f2694bbe700 1 -- 172.21.15.61:0/3972407709 learned_addr learned my addr 172.21.15.61:0/3972407709 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f2694bbe700 1 -- 172.21.15.61:0/3972407709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26900ff140 con 0x7f2690027670 2023-11-16T14:01:30.252 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f2694bbe700 1 --2- 172.21.15.61:0/3972407709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=1927316dd39fef40 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:30.252 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f268f7fe700 1 -- 172.21.15.61:0/3972407709 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268400d600 con 0x7f2690027670 2023-11-16T14:01:30.252 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f268f7fe700 1 -- 172.21.15.61:0/3972407709 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f268400dbf0 con 0x7f2690027670 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.247+0000 7f268f7fe700 1 -- 172.21.15.61:0/3972407709 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2684011bf0 con 0x7f2690027670 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.248+0000 7f2696e22700 1 -- 172.21.15.61:0/3972407709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 msgr2=0x7f26900feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.248+0000 7f2696e22700 1 --2- 172.21.15.61:0/3972407709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).stop 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 -- 172.21.15.61:0/3972407709 shutdown_connections 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 --2- 172.21.15.61:0/3972407709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900feb10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 -- 172.21.15.61:0/3972407709 >> 172.21.15.61:0/3972407709 conn(0x7f26900f6590 msgr2=0x7f26900f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 -- 172.21.15.61:0/3972407709 shutdown_connections 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 -- 172.21.15.61:0/3972407709 wait complete. 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 Processor -- start 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.249+0000 7f2696e22700 1 -- start start 2023-11-16T14:01:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2696e22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:30.254 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2696e22700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26900faa10 con 0x7f2690027670 2023-11-16T14:01:30.254 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2694bbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:30.254 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2694bbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49696/0 (socket says 172.21.15.61:49696) 2023-11-16T14:01:30.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2694bbe700 1 -- 172.21.15.61:0/2725937499 learned_addr learned my addr 172.21.15.61:0/2725937499 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2694bbe700 1 -- 172.21.15.61:0/2725937499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268400b200 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2694bbe700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f2684011de0 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401b410 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f268401ba00 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26900fabb0 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401a770 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.250+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26900fb040 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f268402b070 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2690045bb0 con 0x7f2690027670 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f268d7fa700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 0x7f2674033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f268ffff700 1 -- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 msgr2=0x7f2674033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f2684042c70 con 0x7f2690027670 2023-11-16T14:01:30.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.251+0000 7f268ffff700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 0x7f2674033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T14:01:30.258 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.256+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2684024400 con 0x7f2690027670 2023-11-16T14:01:30.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.452+0000 7f268ffff700 1 -- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 msgr2=0x7f2674033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:30.455 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.452+0000 7f268ffff700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 0x7f2674033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T14:01:30.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.458+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2690110cc0 con 0x7f2690027670 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.459+0000 7f268d7fa700 1 -- 172.21.15.61:0/2725937499 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2684016070 con 0x7f2690027670 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 9, 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "active_name": "smithi061.zcjido", 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-16T14:01:30.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.460+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 msgr2=0x7f2674033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.460+0000 7f2696e22700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 0x7f2674033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.460+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 msgr2=0x7f26900fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.460+0000 7f2696e22700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f2684011de0 tx=0x7f2684006ee0).stop 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 shutdown_connections 2023-11-16T14:01:30.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7f2674031040 0x7f2674033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:30.465 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 --2- 172.21.15.61:0/2725937499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2690027670 0x7f26900fc590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:30.465 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 >> 172.21.15.61:0/2725937499 conn(0x7f26900f6590 msgr2=0x7f26900f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:30.465 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 shutdown_connections 2023-11-16T14:01:30.465 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:30.461+0000 7f2696e22700 1 -- 172.21.15.61:0/2725937499 wait complete. 2023-11-16T14:01:30.686 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:30 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:29.407784+0000 mon.smithi061 (mon.0) 70 : audit [INF] from='client.? 172.21.15.61:0/3546143642' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T14:01:30.686 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:30 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:29.407828+0000 mon.smithi061 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi061.zcjido(active, since 15s) 2023-11-16T14:01:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for the mgr to restart... 2023-11-16T14:01:30.846 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr epoch 9... 2023-11-16T14:01:31.213 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.209+0000 7fccb11e6700 1 Processor -- start 2023-11-16T14:01:31.213 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.209+0000 7fccb11e6700 1 -- start start 2023-11-16T14:01:31.213 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccb11e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccb11e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0ff000 con 0x7fccac027670 2023-11-16T14:01:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccaad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccaad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49706/0 (socket says 172.21.15.61:49706) 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccaad9d700 1 -- 172.21.15.61:0/88729616 learned_addr learned my addr 172.21.15.61:0/88729616 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccaad9d700 1 -- 172.21.15.61:0/88729616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0ff140 con 0x7fccac027670 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fccaad9d700 1 --2- 172.21.15.61:0/88729616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcc9c00b5f0 tx=0x7fcc9c006ee0).ready entity=mon.0 client_cookie=b157c45b2cdc99c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fcca9d9b700 1 -- 172.21.15.61:0/88729616 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c00d600 con 0x7fccac027670 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fcca9d9b700 1 -- 172.21.15.61:0/88729616 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcc9c00dbf0 con 0x7fccac027670 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.210+0000 7fcca9d9b700 1 -- 172.21.15.61:0/88729616 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c011bf0 con 0x7fccac027670 2023-11-16T14:01:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 -- 172.21.15.61:0/88729616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 msgr2=0x7fccac0feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 --2- 172.21.15.61:0/88729616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcc9c00b5f0 tx=0x7fcc9c006ee0).stop 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 -- 172.21.15.61:0/88729616 shutdown_connections 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 --2- 172.21.15.61:0/88729616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac0feb10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 -- 172.21.15.61:0/88729616 >> 172.21.15.61:0/88729616 conn(0x7fccac0f6590 msgr2=0x7fccac0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 -- 172.21.15.61:0/88729616 shutdown_connections 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.211+0000 7fccb11e6700 1 -- 172.21.15.61:0/88729616 wait complete. 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccb11e6700 1 Processor -- start 2023-11-16T14:01:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccb11e6700 1 -- start start 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccb11e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccb11e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac10d390 con 0x7fccac027670 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccaad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccaad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49712/0 (socket says 172.21.15.61:49712) 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccaad9d700 1 -- 172.21.15.61:0/2287693364 learned_addr learned my addr 172.21.15.61:0/2287693364 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccaad9d700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c00b200 con 0x7fccac027670 2023-11-16T14:01:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fccaad9d700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcc9c011de0 tx=0x7fcc9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.212+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c01b410 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcc9c01ba00 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac10d590 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c01a770 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac10da20 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fcc9c02b070 con 0x7fccac027670 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.213+0000 7fcc937fe700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.214+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc98033b70 con 0x7fcc98030ff0 2023-11-16T14:01:31.222 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.214+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fcc9c042c90 con 0x7fccac027670 2023-11-16T14:01:31.222 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.214+0000 7fccaa59c700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:31.222 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.214+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T14:01:31.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.415+0000 7fccaa59c700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:31.418 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.415+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T14:01:31.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:30.460058+0000 mon.smithi061 (mon.0) 72 : audit [DBG] from='client.? 172.21.15.61:0/2725937499' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T14:01:31.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.816+0000 7fccaa59c700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:31.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:31.816+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-16T14:01:32.620 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:32.617+0000 7fccaa59c700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:32.620 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:32.617+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-16T14:01:34.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:34.218+0000 7fccaa59c700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/1639220556 2023-11-16T14:01:34.221 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:34.218+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-16T14:01:34.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:34.830+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7fcc9c00f070 con 0x7fccac027670 2023-11-16T14:01:34.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:34.831+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 msgr2=0x7fcc980334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T14:01:34.834 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:34.831+0000 7fcc937fe700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:35.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:34.776999+0000 mon.smithi061 (mon.0) 73 : cluster [INF] Active manager daemon smithi061.zcjido restarted 2023-11-16T14:01:35.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:34.777400+0000 mon.smithi061 (mon.0) 74 : cluster [INF] Activating manager daemon smithi061.zcjido 2023-11-16T14:01:35.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:34.781625+0000 mon.smithi061 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.835+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7fcc9c01bb70 con 0x7fccac027670 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.835+0000 7fcc937fe700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 0x7fcc98036f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.835+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc98033b70 con 0x7fcc98034b40 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.836+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 0x7fcc98036f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.836+0000 7fccaa59c700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 0x7fcc98036f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcca0006fd0 tx=0x7fcca0006d30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.837+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fcc98033b70 con 0x7fcc98034b40 2023-11-16T14:01:35.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.845+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fccac0591e0 con 0x7fcc98034b40 2023-11-16T14:01:35.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.845+0000 7fcc937fe700 1 -- 172.21.15.61:0/2287693364 <== mgr.14156 v2:172.21.15.61:6800/4132389775 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fccac0591e0 con 0x7fcc98034b40 2023-11-16T14:01:35.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-11-16T14:01:35.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-11-16T14:01:35.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "initialized": true 2023-11-16T14:01:35.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 msgr2=0x7fcc98036f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 0x7fcc98036f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcca0006fd0 tx=0x7fcca0006d30).stop 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 msgr2=0x7fccac10cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcc9c011de0 tx=0x7fcc9c006ee0).stop 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 shutdown_connections 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc98034b40 0x7fcc98036f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccac027670 0x7fccac10cea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 --2- 172.21.15.61:0/2287693364 >> [v2:172.21.15.61:6800/1639220556,v1:172.21.15.61:6801/1639220556] conn(0x7fcc98030ff0 0x7fcc980334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 >> 172.21.15.61:0/2287693364 conn(0x7fccac0f6590 msgr2=0x7fccac0f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 shutdown_connections 2023-11-16T14:01:35.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:35.846+0000 7fccb11e6700 1 -- 172.21.15.61:0/2287693364 wait complete. 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:34.832572+0000 mon.smithi061 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi061.zcjido(active, starting, since 0.0552542s) 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.835809+0000 mon.smithi061 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.836779+0000 mon.smithi061 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mgr metadata", "who": "smithi061.zcjido", "id": "smithi061.zcjido"}]: dispatch 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.837511+0000 mon.smithi061 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.837695+0000 mon.smithi061 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.837836+0000 mon.smithi061 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:34.851469+0000 mon.smithi061 (mon.0) 82 : cluster [INF] Manager daemon smithi061.zcjido is now available 2023-11-16T14:01:36.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.955125+0000 mon.smithi061 (mon.0) 83 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:36.207 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.957397+0000 mon.smithi061 (mon.0) 84 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T14:01:36.207 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.976688+0000 mon.smithi061 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:01:36.207 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:34.978231+0000 mon.smithi061 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:01:36.207 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:35.583276+0000 mon.smithi061 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:mgr epoch 9 is available 2023-11-16T14:01:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:Generating a dashboard self-signed certificate... 2023-11-16T14:01:36.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.765+0000 7f6a917ef700 1 Processor -- start 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.765+0000 7f6a917ef700 1 -- start start 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a917ef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a917ef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0fb710 con 0x7f6a8c0fae00 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a8affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a8affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49774/0 (socket says 172.21.15.61:49774) 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a8affd700 1 -- 172.21.15.61:0/1857307111 learned_addr learned my addr 172.21.15.61:0/1857307111 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.766+0000 7f6a8affd700 1 -- 172.21.15.61:0/1857307111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c0fb850 con 0x7f6a8c0fae00 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a8affd700 1 --2- 172.21.15.61:0/1857307111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6a7c00b5f0 tx=0x7f6a7c006ee0).ready entity=mon.0 client_cookie=5bc4e1874652ec66 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a89ffb700 1 -- 172.21.15.61:0/1857307111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c00d600 con 0x7f6a8c0fae00 2023-11-16T14:01:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a89ffb700 1 -- 172.21.15.61:0/1857307111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6a7c00dbf0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.771 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a89ffb700 1 -- 172.21.15.61:0/1857307111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c011bf0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.771 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 -- 172.21.15.61:0/1857307111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 msgr2=0x7f6a8c0fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:36.771 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 --2- 172.21.15.61:0/1857307111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6a7c00b5f0 tx=0x7f6a7c006ee0).stop 2023-11-16T14:01:36.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 -- 172.21.15.61:0/1857307111 shutdown_connections 2023-11-16T14:01:36.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 --2- 172.21.15.61:0/1857307111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c0fb220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:36.773 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 -- 172.21.15.61:0/1857307111 >> 172.21.15.61:0/1857307111 conn(0x7f6a8c0f6590 msgr2=0x7f6a8c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 -- 172.21.15.61:0/1857307111 shutdown_connections 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.767+0000 7f6a917ef700 1 -- 172.21.15.61:0/1857307111 wait complete. 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a917ef700 1 Processor -- start 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a917ef700 1 -- start start 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a917ef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a917ef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c10d3a0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a8affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a8affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49788/0 (socket says 172.21.15.61:49788) 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.768+0000 7f6a8affd700 1 -- 172.21.15.61:0/953067917 learned_addr learned my addr 172.21.15.61:0/953067917 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a8affd700 1 -- 172.21.15.61:0/953067917 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c00b200 con 0x7f6a8c0fae00 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a8affd700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6a7c006800 tx=0x7f6a7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c00dec0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.774 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6a7c01b8a0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c10d5a0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c01a6f0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.769+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c10da30 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f6a7c02b070 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a837fe700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 0x7f6a6c033400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6a7c042a70 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c045bb0 con 0x7f6a8c0fae00 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a8a7fc700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 0x7f6a6c033400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:36.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.770+0000 7f6a8a7fc700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 0x7f6a6c033400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6a740096a0 tx=0x7f6a74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:36.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.775+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6a7c016070 con 0x7f6a8c0fae00 2023-11-16T14:01:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.842+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f6a7c00f070 con 0x7f6a8c0fae00 2023-11-16T14:01:36.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:36.942+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a8c10e290 con 0x7f6a6c030f40 2023-11-16T14:01:37.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.020+0000 7f6a837fe700 1 -- 172.21.15.61:0/953067917 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f6a8c10e290 con 0x7f6a6c030f40 2023-11-16T14:01:37.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-16T14:01:37.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 msgr2=0x7f6a6c033400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:37.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 0x7f6a6c033400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6a740096a0 tx=0x7f6a74006b40).stop 2023-11-16T14:01:37.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 msgr2=0x7f6a8c10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:37.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6a7c006800 tx=0x7f6a7c006ee0).stop 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 shutdown_connections 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a6c030f40 0x7f6a6c033400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 --2- 172.21.15.61:0/953067917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a8c0fae00 0x7f6a8c10ceb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.021+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 >> 172.21.15.61:0/953067917 conn(0x7f6a8c0f6590 msgr2=0x7f6a8c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.022+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 shutdown_connections 2023-11-16T14:01:37.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.022+0000 7f6a917ef700 1 -- 172.21.15.61:0/953067917 wait complete. 2023-11-16T14:01:37.202 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:35.838136+0000 mgr.smithi061.zcjido (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T14:01:37.202 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:35.838241+0000 mon.smithi061 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi061.zcjido(active, since 1.06092s) 2023-11-16T14:01:37.202 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:35.846649+0000 mgr.smithi061.zcjido (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T14:01:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial admin user... 2023-11-16T14:01:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee559dd700 1 Processor -- start 2023-11-16T14:01:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee559dd700 1 -- start start 2023-11-16T14:01:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee559dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee559dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee500fcda0 con 0x7fee500fc4b0 2023-11-16T14:01:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee4effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee4effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59564/0 (socket says 172.21.15.61:59564) 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee4effd700 1 -- 172.21.15.61:0/4232849060 learned_addr learned my addr 172.21.15.61:0/4232849060 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee4effd700 1 -- 172.21.15.61:0/4232849060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500fcee0 con 0x7fee500fc4b0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.730+0000 7fee4effd700 1 --2- 172.21.15.61:0/4232849060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fee4000b5f0 tx=0x7fee40006ee0).ready entity=mon.0 client_cookie=db54aadb39eb276b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee4dffb700 1 -- 172.21.15.61:0/4232849060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4000d600 con 0x7fee500fc4b0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee4dffb700 1 -- 172.21.15.61:0/4232849060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee4000dbf0 con 0x7fee500fc4b0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee4dffb700 1 -- 172.21.15.61:0/4232849060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee40011bf0 con 0x7fee500fc4b0 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 -- 172.21.15.61:0/4232849060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 msgr2=0x7fee500fc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 --2- 172.21.15.61:0/4232849060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fee4000b5f0 tx=0x7fee40006ee0).stop 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 -- 172.21.15.61:0/4232849060 shutdown_connections 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 --2- 172.21.15.61:0/4232849060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee500fc8b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 -- 172.21.15.61:0/4232849060 >> 172.21.15.61:0/4232849060 conn(0x7fee500f5ce0 msgr2=0x7fee500f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 -- 172.21.15.61:0/4232849060 shutdown_connections 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.731+0000 7fee559dd700 1 -- 172.21.15.61:0/4232849060 wait complete. 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee559dd700 1 Processor -- start 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee559dd700 1 -- start start 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee559dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee559dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5010d370 con 0x7fee500fc4b0 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee4effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee4effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59566/0 (socket says 172.21.15.61:59566) 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.732+0000 7fee4effd700 1 -- 172.21.15.61:0/1636000812 learned_addr learned my addr 172.21.15.61:0/1636000812 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee4effd700 1 -- 172.21.15.61:0/1636000812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4000b200 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee4effd700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fee40006800 tx=0x7fee40006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4000dec0 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5010d570 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5010da00 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee4001b8a0 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.733+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4001a6f0 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fee4002b070 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee377fe700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 0x7fee3c033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fee40042a60 con 0x7fee500fc4b0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee4e7fc700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 0x7fee3c033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee4e7fc700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 0x7fee3c033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fee440096a0 tx=0x7fee44006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.734+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee30004fa0 con 0x7fee500fc4b0 2023-11-16T14:01:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.739+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fee40016070 con 0x7fee500fc4b0 2023-11-16T14:01:37.918 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:37.916+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fee30005bf0 con 0x7fee3c031040 2023-11-16T14:01:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.161+0000 7fee377fe700 1 -- 172.21.15.61:0/1636000812 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fee30005bf0 con 0x7fee3c031040 2023-11-16T14:01:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$FEtcrM1vPRtf5oK.5Lxh2OTUsrhiVZNKkBT2ws/2VwLSFH3N4v/my", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700143298, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 msgr2=0x7fee3c033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 0x7fee3c033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fee440096a0 tx=0x7fee44006b40).stop 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 msgr2=0x7fee5010ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fee40006800 tx=0x7fee40006ee0).stop 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 shutdown_connections 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fee3c031040 0x7fee3c033500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:38.166 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 --2- 172.21.15.61:0/1636000812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee500fc4b0 0x7fee5010ce80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:38.167 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 >> 172.21.15.61:0/1636000812 conn(0x7fee500f5ce0 msgr2=0x7fee500f77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:38.167 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 shutdown_connections 2023-11-16T14:01:38.167 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.163+0000 7fee559dd700 1 -- 172.21.15.61:0/1636000812 wait complete. 2023-11-16T14:01:38.196 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:36.844286+0000 mon.smithi061 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi061.zcjido(active, since 2s) 2023-11-16T14:01:38.196 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:36.943762+0000 mgr.smithi061.zcjido (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:38.196 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:37.018372+0000 mon.smithi061 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:38.196 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:37.020850+0000 mon.smithi061 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:38.528 INFO:teuthology.orchestra.run.smithi061.stderr:Fetching dashboard port number... 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b70575700 1 Processor -- start 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b70575700 1 -- start start 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b70575700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b70575700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680ff000 con 0x7f2b6806ade0 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b6e311700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b6e311700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59570/0 (socket says 172.21.15.61:59570) 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.873+0000 7f2b6e311700 1 -- 172.21.15.61:0/3833136951 learned_addr learned my addr 172.21.15.61:0/3833136951 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.874+0000 7f2b6e311700 1 -- 172.21.15.61:0/3833136951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680ff140 con 0x7f2b6806ade0 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.874+0000 7f2b6e311700 1 --2- 172.21.15.61:0/3833136951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2b5c00b5f0 tx=0x7f2b5c006ee0).ready entity=mon.0 client_cookie=39a1a701e1310afe server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.874+0000 7f2b6d30f700 1 -- 172.21.15.61:0/3833136951 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c00d600 con 0x7f2b6806ade0 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.874+0000 7f2b6d30f700 1 -- 172.21.15.61:0/3833136951 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2b5c00dbf0 con 0x7f2b6806ade0 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.874+0000 7f2b6d30f700 1 -- 172.21.15.61:0/3833136951 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c011bf0 con 0x7f2b6806ade0 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- 172.21.15.61:0/3833136951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 msgr2=0x7f2b680feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 --2- 172.21.15.61:0/3833136951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2b5c00b5f0 tx=0x7f2b5c006ee0).stop 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- 172.21.15.61:0/3833136951 shutdown_connections 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 --2- 172.21.15.61:0/3833136951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680feb10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- 172.21.15.61:0/3833136951 >> 172.21.15.61:0/3833136951 conn(0x7f2b680f6590 msgr2=0x7f2b680f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- 172.21.15.61:0/3833136951 shutdown_connections 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- 172.21.15.61:0/3833136951 wait complete. 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 Processor -- start 2023-11-16T14:01:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.875+0000 7f2b70575700 1 -- start start 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b70575700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b70575700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680faa10 con 0x7f2b6806ade0 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b6e311700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b6e311700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59572/0 (socket says 172.21.15.61:59572) 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b6e311700 1 -- 172.21.15.61:0/3315634143 learned_addr learned my addr 172.21.15.61:0/3315634143 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b6e311700 1 -- 172.21.15.61:0/3315634143 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c00b200 con 0x7f2b6806ade0 2023-11-16T14:01:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b6e311700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f2b5c006800 tx=0x7f2b5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c00dec0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2b5c01b8a0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b680fabb0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c01a7b0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.876+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b680fb040 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.877+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f2b5c02b070 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.877+0000 7f2b5affd700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 0x7f2b50033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.877+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68045bb0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.877+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2b5c042be0 con 0x7f2b6806ade0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.877+0000 7f2b6db10700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 0x7f2b50033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.878+0000 7f2b6db10700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 0x7f2b50033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2b640096a0 tx=0x7f2b64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:38.884 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:38.882+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2b5c016020 con 0x7f2b6806ade0 2023-11-16T14:01:39.046 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.043+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2b680fff00 con 0x7f2b6806ade0 2023-11-16T14:01:39.047 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.043+0000 7f2b5affd700 1 -- 172.21.15.61:0/3315634143 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2b5c01f070 con 0x7f2b6806ade0 2023-11-16T14:01:39.047 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 8443 2023-11-16T14:01:39.049 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 msgr2=0x7f2b50033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:39.050 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 0x7f2b50033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2b640096a0 tx=0x7f2b64006b40).stop 2023-11-16T14:01:39.050 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 msgr2=0x7f2b680fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:39.050 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f2b5c006800 tx=0x7f2b5c006ee0).stop 2023-11-16T14:01:39.050 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 shutdown_connections 2023-11-16T14:01:39.050 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2b50031040 0x7f2b50033500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:39.051 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 --2- 172.21.15.61:0/3315634143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b6806ade0 0x7f2b680fc590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:39.051 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 >> 172.21.15.61:0/3315634143 conn(0x7f2b680f6590 msgr2=0x7f2b680f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:39.051 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 shutdown_connections 2023-11-16T14:01:39.051 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.045+0000 7f2b70575700 1 -- 172.21.15.61:0/3315634143 wait complete. 2023-11-16T14:01:39.330 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:37.917715+0000 mgr.smithi061.zcjido (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:39.330 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:38.162335+0000 mon.smithi061 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:39.330 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:39.044715+0000 mon.smithi061 (mon.0) 93 : audit [DBG] from='client.? 172.21.15.61:0/3315634143' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T14:01:39.403 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-11-16T14:01:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-11-16T14:01:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-11-16T14:01:39.412 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T14:01:39.414 INFO:teuthology.orchestra.run.smithi061.stderr:Ceph Dashboard is now available at: 2023-11-16T14:01:39.414 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:39.415 INFO:teuthology.orchestra.run.smithi061.stderr: URL: https://smithi061.front.sepia.ceph.com:8443/ 2023-11-16T14:01:39.415 INFO:teuthology.orchestra.run.smithi061.stderr: User: admin 2023-11-16T14:01:39.415 INFO:teuthology.orchestra.run.smithi061.stderr: Password: pj77i3mdjm 2023-11-16T14:01:39.415 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:39.415 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling autotune for osd_memory_target 2023-11-16T14:01:39.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f352333f700 1 Processor -- start 2023-11-16T14:01:39.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f352333f700 1 -- start start 2023-11-16T14:01:39.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f352333f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:39.775 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f352333f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0fde80 con 0x7f351c0fd570 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f35210db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f35210db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59576/0 (socket says 172.21.15.61:59576) 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.770+0000 7f35210db700 1 -- 172.21.15.61:0/452991280 learned_addr learned my addr 172.21.15.61:0/452991280 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f35210db700 1 -- 172.21.15.61:0/452991280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0fdfc0 con 0x7f351c0fd570 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f35210db700 1 --2- 172.21.15.61:0/452991280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f351800b5f0 tx=0x7f3518006ee0).ready entity=mon.0 client_cookie=52c9c06cabca8061 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f350bfff700 1 -- 172.21.15.61:0/452991280 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351800d600 con 0x7f351c0fd570 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f350bfff700 1 -- 172.21.15.61:0/452991280 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f351800dbf0 con 0x7f351c0fd570 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f350bfff700 1 -- 172.21.15.61:0/452991280 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3518011bf0 con 0x7f351c0fd570 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f352333f700 1 -- 172.21.15.61:0/452991280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 msgr2=0x7f351c0fd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f352333f700 1 --2- 172.21.15.61:0/452991280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f351800b5f0 tx=0x7f3518006ee0).stop 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f352333f700 1 -- 172.21.15.61:0/452991280 shutdown_connections 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f352333f700 1 --2- 172.21.15.61:0/452991280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c0fd990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.771+0000 7f352333f700 1 -- 172.21.15.61:0/452991280 >> 172.21.15.61:0/452991280 conn(0x7f351c0f5d10 msgr2=0x7f351c0f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 -- 172.21.15.61:0/452991280 shutdown_connections 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 -- 172.21.15.61:0/452991280 wait complete. 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 Processor -- start 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 -- start start 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.772+0000 7f352333f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c10d3c0 con 0x7f351c0fd570 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f35210db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:39.777 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f35210db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59592/0 (socket says 172.21.15.61:59592) 2023-11-16T14:01:39.778 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f35210db700 1 -- 172.21.15.61:0/3360231815 learned_addr learned my addr 172.21.15.61:0/3360231815 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f35210db700 1 -- 172.21.15.61:0/3360231815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351800b200 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f35210db700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f3518011de0 tx=0x7f3518006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351801b410 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f351801ba00 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c10d5c0 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351801a770 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.773+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c10da50 con 0x7f351c0fd570 2023-11-16T14:01:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.774+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c045bb0 con 0x7f351c0fd570 2023-11-16T14:01:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.775+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f351802b070 con 0x7f351c0fd570 2023-11-16T14:01:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.775+0000 7f3509ffb700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 0x7f3504033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.775+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3518043380 con 0x7f351c0fd570 2023-11-16T14:01:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.775+0000 7f35208da700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 0x7f3504033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.776+0000 7f35208da700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 0x7f3504033500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.779+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3518023ba0 con 0x7f351c0fd570 2023-11-16T14:01:39.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.939+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f351c0fa920 con 0x7f351c0fd570 2023-11-16T14:01:39.942 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.939+0000 7f3509ffb700 1 -- 172.21.15.61:0/3360231815 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7f35180320c0 con 0x7f351c0fd570 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 msgr2=0x7f3504033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 0x7f3504033500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).stop 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 msgr2=0x7f351c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f3518011de0 tx=0x7f3518006ee0).stop 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 shutdown_connections 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3504031040 0x7f3504033500 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:39.944 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 --2- 172.21.15.61:0/3360231815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f351c0fd570 0x7f351c10ced0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:39.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 >> 172.21.15.61:0/3360231815 conn(0x7f351c0f5d10 msgr2=0x7f351c0fb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:39.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 shutdown_connections 2023-11-16T14:01:39.945 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:39.941+0000 7f352333f700 1 -- 172.21.15.61:0/3360231815 wait complete. 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c83f87700 1 Processor -- start 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c83f87700 1 -- start start 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c83f87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c83f87700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0fb710 con 0x7f3c7c0fae00 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c81d23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c81d23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59600/0 (socket says 172.21.15.61:59600) 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.665+0000 7f3c81d23700 1 -- 172.21.15.61:0/2744985302 learned_addr learned my addr 172.21.15.61:0/2744985302 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c81d23700 1 -- 172.21.15.61:0/2744985302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0fb850 con 0x7f3c7c0fae00 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c81d23700 1 --2- 172.21.15.61:0/2744985302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3c7800b5f0 tx=0x7f3c78006ee0).ready entity=mon.0 client_cookie=53a833d241250ec2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c80d21700 1 -- 172.21.15.61:0/2744985302 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7800d600 con 0x7f3c7c0fae00 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c80d21700 1 -- 172.21.15.61:0/2744985302 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3c7800dbf0 con 0x7f3c7c0fae00 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c80d21700 1 -- 172.21.15.61:0/2744985302 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c78011bf0 con 0x7f3c7c0fae00 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c83f87700 1 -- 172.21.15.61:0/2744985302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 msgr2=0x7f3c7c0fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.666+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2744985302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3c7800b5f0 tx=0x7f3c78006ee0).stop 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 -- 172.21.15.61:0/2744985302 shutdown_connections 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2744985302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c0fb220 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 -- 172.21.15.61:0/2744985302 >> 172.21.15.61:0/2744985302 conn(0x7f3c7c0f6590 msgr2=0x7f3c7c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 -- 172.21.15.61:0/2744985302 shutdown_connections 2023-11-16T14:01:40.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 -- 172.21.15.61:0/2744985302 wait complete. 2023-11-16T14:01:40.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 Processor -- start 2023-11-16T14:01:40.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.667+0000 7f3c83f87700 1 -- start start 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c83f87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c83f87700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c108f90 con 0x7f3c7c0fae00 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c81d23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c81d23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59614/0 (socket says 172.21.15.61:59614) 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c81d23700 1 -- 172.21.15.61:0/2908075896 learned_addr learned my addr 172.21.15.61:0/2908075896 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c81d23700 1 -- 172.21.15.61:0/2908075896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7800b200 con 0x7f3c7c0fae00 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c81d23700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3c78006800 tx=0x7f3c78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7800dec0 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c109190 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3c7801b8a0 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7801a690 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.668+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c109620 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.669+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f3c7802b070 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.670+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c045bb0 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.670+0000 7f3c6a7fc700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 0x7f3c64035700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.670+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3c78042c30 con 0x7f3c7c0fae00 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.670+0000 7f3c81522700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 0x7f3c64035700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.670+0000 7f3c81522700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 0x7f3c64035700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f3c6c0096a0 tx=0x7f3c6c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:40.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.674+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3c7801ba10 con 0x7f3c7c0fae00 2023-11-16T14:01:40.855 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.852+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3c7800f070 con 0x7f3c7c0fae00 2023-11-16T14:01:40.923 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.920+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f3c7c10ac40 con 0x7f3c7c0fae00 2023-11-16T14:01:40.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.922+0000 7f3c6a7fc700 1 -- 172.21.15.61:0/2908075896 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v31)=0 set mgr/dashboard/cluster/status v31) v1 ==== 153+0+0 (secure 0 0 0) 0x7f3c78016070 con 0x7f3c7c0fae00 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 msgr2=0x7f3c64035700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 0x7f3c64035700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f3c6c0096a0 tx=0x7f3c6c006b40).stop 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 msgr2=0x7f3c7c108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3c78006800 tx=0x7f3c78006ee0).stop 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 shutdown_connections 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c64033240 0x7f3c64035700 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 --2- 172.21.15.61:0/2908075896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0fae00 0x7f3c7c108aa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.924+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 >> 172.21.15.61:0/2908075896 conn(0x7f3c7c0f6590 msgr2=0x7f3c7c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.925+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 shutdown_connections 2023-11-16T14:01:40.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-11-16T14:01:40.925+0000 7f3c83f87700 1 -- 172.21.15.61:0/2908075896 wait complete. 2023-11-16T14:01:40.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:Or, if you are only running a single cluster on this host: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.279 INFO:teuthology.orchestra.run.smithi061.stderr: ceph telemetry on 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:For more information see: 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:01:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:Bootstrap complete. 2023-11-16T14:01:41.314 INFO:tasks.cephadm:Fetching config... 2023-11-16T14:01:41.314 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:01:41.314 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T14:01:41.373 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T14:01:41.373 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:01:41.373 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T14:01:41.433 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T14:01:41.433 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:01:41.433 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/mon.smithi061/keyring of=/dev/stdout 2023-11-16T14:01:41.503 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T14:01:41.503 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-11-16T14:01:41.503 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T14:01:41.564 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T14:01:41.564 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuAqycKkdidqQUDUnpngVlA1vZPUjRvXU+DFGxfVKi0R4AuVJYW/ytKPIOL8scDKOuaoFVCKAd+6SV9JYSO9SDI2RnkfnRDFwK6JmkdiIxS5uW0f864dm7dVGUjv1N3LRGPbQaw5ZB18z4u/R7DTkr4CmhxgAplebbIaUi8vJTRY0xSTB7xNLHzd1BJGMmGOSuemqxsK5flZSMjYg4jRXxJWAzapxCV/8i4/3v8ViTFAJ9OIYedaFanx+n61zY47OeRKGhcLaA7NUaWUTfH2bijUgaOJWGhjNCttD6+ahTeVWOyGZfqi68SNmf7f4TWq96hLltY2J+TG+9KYKApaaugfzQyndBYTbX+nXwKN4j3yZhpD/XFLEbbUIzD9lO8czfmbiyIT6c2OJwVD232zyULYmXCnIQI5Sd3c2BdRouXBvt9nYkCrRA3VralX3JgtjxOBX/Si6HGzwkexHoGNcBakR+Uk01IGshdG9xpbEafX3B869yYG+Qx6yXaaH8rzE= ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T14:01:41.643 INFO:teuthology.orchestra.run.smithi061.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuAqycKkdidqQUDUnpngVlA1vZPUjRvXU+DFGxfVKi0R4AuVJYW/ytKPIOL8scDKOuaoFVCKAd+6SV9JYSO9SDI2RnkfnRDFwK6JmkdiIxS5uW0f864dm7dVGUjv1N3LRGPbQaw5ZB18z4u/R7DTkr4CmhxgAplebbIaUi8vJTRY0xSTB7xNLHzd1BJGMmGOSuemqxsK5flZSMjYg4jRXxJWAzapxCV/8i4/3v8ViTFAJ9OIYedaFanx+n61zY47OeRKGhcLaA7NUaWUTfH2bijUgaOJWGhjNCttD6+ahTeVWOyGZfqi68SNmf7f4TWq96hLltY2J+TG+9KYKApaaugfzQyndBYTbX+nXwKN4j3yZhpD/XFLEbbUIzD9lO8czfmbiyIT6c2OJwVD232zyULYmXCnIQI5Sd3c2BdRouXBvt9nYkCrRA3VralX3JgtjxOBX/Si6HGzwkexHoGNcBakR+Uk01IGshdG9xpbEafX3B869yYG+Qx6yXaaH8rzE= ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:01:41.655 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuAqycKkdidqQUDUnpngVlA1vZPUjRvXU+DFGxfVKi0R4AuVJYW/ytKPIOL8scDKOuaoFVCKAd+6SV9JYSO9SDI2RnkfnRDFwK6JmkdiIxS5uW0f864dm7dVGUjv1N3LRGPbQaw5ZB18z4u/R7DTkr4CmhxgAplebbIaUi8vJTRY0xSTB7xNLHzd1BJGMmGOSuemqxsK5flZSMjYg4jRXxJWAzapxCV/8i4/3v8ViTFAJ9OIYedaFanx+n61zY47OeRKGhcLaA7NUaWUTfH2bijUgaOJWGhjNCttD6+ahTeVWOyGZfqi68SNmf7f4TWq96hLltY2J+TG+9KYKApaaugfzQyndBYTbX+nXwKN4j3yZhpD/XFLEbbUIzD9lO8czfmbiyIT6c2OJwVD232zyULYmXCnIQI5Sd3c2BdRouXBvt9nYkCrRA3VralX3JgtjxOBX/Si6HGzwkexHoGNcBakR+Uk01IGshdG9xpbEafX3B869yYG+Qx6yXaaH8rzE= ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T14:01:41.701 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuAqycKkdidqQUDUnpngVlA1vZPUjRvXU+DFGxfVKi0R4AuVJYW/ytKPIOL8scDKOuaoFVCKAd+6SV9JYSO9SDI2RnkfnRDFwK6JmkdiIxS5uW0f864dm7dVGUjv1N3LRGPbQaw5ZB18z4u/R7DTkr4CmhxgAplebbIaUi8vJTRY0xSTB7xNLHzd1BJGMmGOSuemqxsK5flZSMjYg4jRXxJWAzapxCV/8i4/3v8ViTFAJ9OIYedaFanx+n61zY47OeRKGhcLaA7NUaWUTfH2bijUgaOJWGhjNCttD6+ahTeVWOyGZfqi68SNmf7f4TWq96hLltY2J+TG+9KYKApaaugfzQyndBYTbX+nXwKN4j3yZhpD/XFLEbbUIzD9lO8czfmbiyIT6c2OJwVD232zyULYmXCnIQI5Sd3c2BdRouXBvt9nYkCrRA3VralX3JgtjxOBX/Si6HGzwkexHoGNcBakR+Uk01IGshdG9xpbEafX3B869yYG+Qx6yXaaH8rzE= ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7 2023-11-16T14:01:41.712 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T14:01:42.226 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:40.854791+0000 mon.smithi061 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi061.zcjido(active, since 6s) 2023-11-16T14:01:42.226 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:40.923908+0000 mon.smithi061 (mon.0) 95 : audit [INF] from='client.? 172.21.15.61:0/2908075896' entity='client.admin' 2023-11-16T14:01:43.301 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- 172.21.15.61:0/447313227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 msgr2=0x7f2cec0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:43.302 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 --2- 172.21.15.61:0/447313227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec0f9160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f2cdc009fb0 tx=0x7f2cdc009b30).stop 2023-11-16T14:01:43.302 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- 172.21.15.61:0/447313227 shutdown_connections 2023-11-16T14:01:43.302 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 --2- 172.21.15.61:0/447313227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec0f9160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:43.302 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- 172.21.15.61:0/447313227 >> 172.21.15.61:0/447313227 conn(0x7f2cec0f0be0 msgr2=0x7f2cec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:43.302 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- 172.21.15.61:0/447313227 shutdown_connections 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- 172.21.15.61:0/447313227 wait complete. 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 Processor -- start 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.296+0000 7f2cf246c700 1 -- start start 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cf246c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cf246c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec106c40 con 0x7f2cec060360 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:43.303 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59632/0 (socket says 172.21.15.61:59632) 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cebfff700 1 -- 172.21.15.61:0/400647112 learned_addr learned my addr 172.21.15.61:0/400647112 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cebfff700 1 -- 172.21.15.61:0/400647112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc005040 con 0x7f2cec060360 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cebfff700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2cdc009f80 tx=0x7f2cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cdc009320 con 0x7f2cec060360 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2cdc013680 con 0x7f2cec060360 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec106e40 con 0x7f2cec060360 2023-11-16T14:01:43.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cdc01eda0 con 0x7f2cec060360 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.297+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec1072d0 con 0x7f2cec060360 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.298+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2cdc014420 con 0x7f2cec060360 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.298+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec045bb0 con 0x7f2cec060360 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.298+0000 7f2ce8ff9700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 0x7f2cd4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.299+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2cdc0423c0 con 0x7f2cec060360 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.299+0000 7f2ceb7fe700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 0x7f2cd4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:43.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.299+0000 7f2ceb7fe700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 0x7f2cd4033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2ce00096a0 tx=0x7f2ce0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:43.306 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.303+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2cdc01c6e0 con 0x7f2cec060360 2023-11-16T14:01:43.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.465+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2cec0fc060 con 0x7f2cec060360 2023-11-16T14:01:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.469+0000 7f2ce8ff9700 1 -- 172.21.15.61:0/400647112 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2cdc01c320 con 0x7f2cec060360 2023-11-16T14:01:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 msgr2=0x7f2cd4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 0x7f2cd4033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2ce00096a0 tx=0x7f2ce0006b40).stop 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 msgr2=0x7f2cec106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2cdc009f80 tx=0x7f2cdc00b040).stop 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 shutdown_connections 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cd4031090 0x7f2cd4033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 --2- 172.21.15.61:0/400647112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cec060360 0x7f2cec106750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.471+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 >> 172.21.15.61:0/400647112 conn(0x7f2cec0f0be0 msgr2=0x7f2cec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.472+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 shutdown_connections 2023-11-16T14:01:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:43.472+0000 7f2cf246c700 1 -- 172.21.15.61:0/400647112 wait complete. 2023-11-16T14:01:43.826 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-16T14:01:43.826 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-16T14:01:44.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:44 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:43.470266+0000 mon.smithi061 (mon.0) 96 : audit [INF] from='client.? 172.21.15.61:0/400647112' entity='client.admin' 2023-11-16T14:01:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 -- 172.21.15.61:0/3677068439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 msgr2=0x7fdb900f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3677068439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb900f9160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdb84009fb0 tx=0x7fdb84009b30).stop 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 -- 172.21.15.61:0/3677068439 shutdown_connections 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3677068439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb900f9160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 -- 172.21.15.61:0/3677068439 >> 172.21.15.61:0/3677068439 conn(0x7fdb900f0be0 msgr2=0x7fdb900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 -- 172.21.15.61:0/3677068439 shutdown_connections 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.052+0000 7fdb96e15700 1 -- 172.21.15.61:0/3677068439 wait complete. 2023-11-16T14:01:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb96e15700 1 Processor -- start 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb96e15700 1 -- start start 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb96e15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb96e15700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90108e60 con 0x7fdb90060360 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb94bb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb94bb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59660/0 (socket says 172.21.15.61:59660) 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb94bb1700 1 -- 172.21.15.61:0/3256699794 learned_addr learned my addr 172.21.15.61:0/3256699794 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb94bb1700 1 -- 172.21.15.61:0/3256699794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84005040 con 0x7fdb90060360 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.053+0000 7fdb94bb1700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fdb84000c00 tx=0x7fdb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb840095f0 con 0x7fdb90060360 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb90109060 con 0x7fdb90060360 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdb84007e50 con 0x7fdb90060360 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb84014e70 con 0x7fdb90060360 2023-11-16T14:01:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb901094f0 con 0x7fdb90060360 2023-11-16T14:01:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.054+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdb8401d4e0 con 0x7fdb90060360 2023-11-16T14:01:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.055+0000 7fdb8d7fa700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 0x7fdb80033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.055+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb90045bb0 con 0x7fdb90060360 2023-11-16T14:01:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.055+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdb84041d50 con 0x7fdb90060360 2023-11-16T14:01:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.055+0000 7fdb8ffff700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 0x7fdb80033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.055+0000 7fdb8ffff700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 0x7fdb80033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fdb7c0096a0 tx=0x7fdb7c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.059+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdb8401b4a0 con 0x7fdb90060360 2023-11-16T14:01:45.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.224+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb900f1d30 con 0x7fdb80031090 2023-11-16T14:01:45.230 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.228+0000 7fdb8d7fa700 1 -- 172.21.15.61:0/3256699794 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdb900f1d30 con 0x7fdb80031090 2023-11-16T14:01:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 msgr2=0x7fdb80033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 0x7fdb80033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fdb7c0096a0 tx=0x7fdb7c006b40).stop 2023-11-16T14:01:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 msgr2=0x7fdb90108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fdb84000c00 tx=0x7fdb8400b040).stop 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 shutdown_connections 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdb80031090 0x7fdb80033550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 --2- 172.21.15.61:0/3256699794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb90060360 0x7fdb90108970 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 >> 172.21.15.61:0/3256699794 conn(0x7fdb900f0be0 msgr2=0x7fdb900f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 shutdown_connections 2023-11-16T14:01:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:45.230+0000 7fdb96e15700 1 -- 172.21.15.61:0/3256699794 wait complete. 2023-11-16T14:01:45.561 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2023-11-16T14:01:45.561 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:01:45.562 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2023-11-16T14:01:45.584 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-11-16T14:01:45.584 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T14:01:45.645 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2023-11-16T14:01:45.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi112 2023-11-16T14:01:45.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:44.802241+0000 mon.smithi061 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:45.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:45.228900+0000 mon.smithi061 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:45.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:45.283126+0000 mon.smithi061 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config rm", "who": "osd/host:smithi061", "name": "osd_memory_target"}]: dispatch 2023-11-16T14:01:45.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:45.287166+0000 mon.smithi061 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:45.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:45.294542+0000 mgr.smithi061.zcjido (mgr.14156) 6 : cephadm [INF] Deploying daemon alertmanager.smithi061 on smithi061 2023-11-16T14:01:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.716+0000 7fc858689700 1 -- 172.21.15.61:0/4268567977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 msgr2=0x7fc8500f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.716+0000 7fc858689700 1 --2- 172.21.15.61:0/4268567977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc8500f4d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc84c009fb0 tx=0x7fc84c009b30).stop 2023-11-16T14:01:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- 172.21.15.61:0/4268567977 shutdown_connections 2023-11-16T14:01:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 --2- 172.21.15.61:0/4268567977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc8500f4d00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- 172.21.15.61:0/4268567977 >> 172.21.15.61:0/4268567977 conn(0x7fc8500f0360 msgr2=0x7fc8500f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- 172.21.15.61:0/4268567977 shutdown_connections 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- 172.21.15.61:0/4268567977 wait complete. 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 Processor -- start 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- start start 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.717+0000 7fc858689700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc850106e70 con 0x7fc8500f48e0 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc856425700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc856425700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59678/0 (socket says 172.21.15.61:59678) 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc856425700 1 -- 172.21.15.61:0/2044194580 learned_addr learned my addr 172.21.15.61:0/2044194580 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc856425700 1 -- 172.21.15.61:0/2044194580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c005040 con 0x7fc8500f48e0 2023-11-16T14:01:46.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc856425700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fc84c009f80 tx=0x7fc84c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84c009320 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc84c013680 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84c01eda0 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc850107070 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.718+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc850107500 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.719+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850045bb0 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.719+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc84c01e420 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.719+0000 7fc846ffd700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 0x7fc83c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.719+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc84c042350 con 0x7fc8500f48e0 2023-11-16T14:01:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.719+0000 7fc855c24700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 0x7fc83c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:46.724 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.720+0000 7fc855c24700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 0x7fc83c0335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc8400096a0 tx=0x7fc840006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:46.726 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.724+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc84c01e6b0 con 0x7fc8500f48e0 2023-11-16T14:01:46.892 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:46.890+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8500f1530 con 0x7fc83c0310e0 2023-11-16T14:01:47.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.522+0000 7fc846ffd700 1 -- 172.21.15.61:0/2044194580 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc8500f1530 con 0x7fc83c0310e0 2023-11-16T14:01:47.525 INFO:teuthology.orchestra.run.smithi061.stdout:Added host 'smithi112' with addr '172.21.15.112' 2023-11-16T14:01:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.523+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 msgr2=0x7fc83c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.523+0000 7fc858689700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 0x7fc83c0335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc8400096a0 tx=0x7fc840006b40).stop 2023-11-16T14:01:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.523+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 msgr2=0x7fc850106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.523+0000 7fc858689700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fc84c009f80 tx=0x7fc84c00b040).stop 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 shutdown_connections 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc83c0310e0 0x7fc83c0335a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 --2- 172.21.15.61:0/2044194580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8500f48e0 0x7fc850106980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 >> 172.21.15.61:0/2044194580 conn(0x7fc8500f0360 msgr2=0x7fc8500f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 shutdown_connections 2023-11-16T14:01:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:47.524+0000 7fc858689700 1 -- 172.21.15.61:0/2044194580 wait complete. 2023-11-16T14:01:47.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:46.891710+0000 mgr.smithi061.zcjido (mgr.14156) 7 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:47.881 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T14:01:48.859 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:47.522449+0000 mon.smithi061 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:48.859 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:47.522756+0000 mgr.smithi061.zcjido (mgr.14156) 8 : cephadm [INF] Added host smithi112 2023-11-16T14:01:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 -- 172.21.15.61:0/2992399838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 msgr2=0x7fdda00f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 --2- 172.21.15.61:0/2992399838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda00f4d30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdd9c009fb0 tx=0x7fdd9c009b30).stop 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 -- 172.21.15.61:0/2992399838 shutdown_connections 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 --2- 172.21.15.61:0/2992399838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda00f4d30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 -- 172.21.15.61:0/2992399838 >> 172.21.15.61:0/2992399838 conn(0x7fdda00f0390 msgr2=0x7fdda00f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 -- 172.21.15.61:0/2992399838 shutdown_connections 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.040+0000 7fdda874f700 1 -- 172.21.15.61:0/2992399838 wait complete. 2023-11-16T14:01:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda874f700 1 Processor -- start 2023-11-16T14:01:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda874f700 1 -- start start 2023-11-16T14:01:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda874f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda874f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda0106c80 con 0x7fdda00f4910 2023-11-16T14:01:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda64eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda64eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33322/0 (socket says 172.21.15.61:33322) 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda64eb700 1 -- 172.21.15.61:0/3663465252 learned_addr learned my addr 172.21.15.61:0/3663465252 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda64eb700 1 -- 172.21.15.61:0/3663465252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c005040 con 0x7fdda00f4910 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.041+0000 7fdda64eb700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdd9c009f80 tx=0x7fdd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c009320 con 0x7fdda00f4910 2023-11-16T14:01:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdd9c013680 con 0x7fdda00f4910 2023-11-16T14:01:49.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda0106e80 con 0x7fdda00f4910 2023-11-16T14:01:49.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c01eda0 con 0x7fdda00f4910 2023-11-16T14:01:49.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda0107310 con 0x7fdda00f4910 2023-11-16T14:01:49.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.042+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdd9c01e420 con 0x7fdda00f4910 2023-11-16T14:01:49.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.043+0000 7fdd96ffd700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 0x7fdd8c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:49.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.043+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdd9c042300 con 0x7fdda00f4910 2023-11-16T14:01:49.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.043+0000 7fdda5cea700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 0x7fdd8c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:49.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.043+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda00591e0 con 0x7fdda00f4910 2023-11-16T14:01:49.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.043+0000 7fdda5cea700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 0x7fdd8c0335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fdd900096a0 tx=0x7fdd90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:49.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.047+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdd9c01cb10 con 0x7fdda00f4910 2023-11-16T14:01:49.212 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.210+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdda00f15a0 con 0x7fdd8c0310e0 2023-11-16T14:01:49.213 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.211+0000 7fdd96ffd700 1 -- 172.21.15.61:0/3663465252 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fdda00f15a0 con 0x7fdd8c0310e0 2023-11-16T14:01:49.214 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-11-16T14:01:49.214 INFO:teuthology.orchestra.run.smithi061.stdout:[{"addr": "172.21.15.61", "hostname": "smithi061", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}] 2023-11-16T14:01:49.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.212+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 msgr2=0x7fdd8c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:49.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.212+0000 7fdda874f700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 0x7fdd8c0335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fdd900096a0 tx=0x7fdd90006b40).stop 2023-11-16T14:01:49.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.212+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 msgr2=0x7fdda0106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:49.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.212+0000 7fdda874f700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdd9c009f80 tx=0x7fdd9c00b040).stop 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 shutdown_connections 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdd8c0310e0 0x7fdd8c0335a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 --2- 172.21.15.61:0/3663465252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdda00f4910 0x7fdda0106790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 >> 172.21.15.61:0/3663465252 conn(0x7fdda00f0390 msgr2=0x7fdda00f0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 shutdown_connections 2023-11-16T14:01:49.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:49.213+0000 7fdda874f700 1 -- 172.21.15.61:0/3663465252 wait complete. 2023-11-16T14:01:49.581 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T14:01:49.581 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T14:01:49.924 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:49.211898+0000 mgr.smithi061.zcjido (mgr.14156) 9 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T14:01:50.705 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 -- 172.21.15.61:0/2733216515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 msgr2=0x7f8e580f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:50.705 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 --2- 172.21.15.61:0/2733216515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e580f56f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8e48009fb0 tx=0x7f8e48009b30).stop 2023-11-16T14:01:50.706 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 -- 172.21.15.61:0/2733216515 shutdown_connections 2023-11-16T14:01:50.706 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 --2- 172.21.15.61:0/2733216515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e580f56f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:50.706 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 -- 172.21.15.61:0/2733216515 >> 172.21.15.61:0/2733216515 conn(0x7f8e580f0a30 msgr2=0x7f8e580f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:50.706 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 -- 172.21.15.61:0/2733216515 shutdown_connections 2023-11-16T14:01:50.706 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.701+0000 7f8e5759e700 1 -- 172.21.15.61:0/2733216515 wait complete. 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5759e700 1 Processor -- start 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5759e700 1 -- start start 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58060360 con 0x7f8e580f52d0 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33346/0 (socket says 172.21.15.61:33346) 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.702+0000 7f8e5659c700 1 -- 172.21.15.61:0/1909773640 learned_addr learned my addr 172.21.15.61:0/1909773640 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e5659c700 1 -- 172.21.15.61:0/1909773640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e48005040 con 0x7f8e580f52d0 2023-11-16T14:01:50.707 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e5659c700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8e480089f0 tx=0x7f8e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:50.708 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e48009320 con 0x7f8e580f52d0 2023-11-16T14:01:50.708 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e58060560 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e48013ec0 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e48014d30 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.703+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e58109a00 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.704+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8e480263a0 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.704+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e58045bb0 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.704+0000 7f8e46ffd700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 0x7f8e40033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.704+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e480423a0 con 0x7f8e580f52d0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.704+0000 7f8e55d9b700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 0x7f8e40033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.705+0000 7f8e55d9b700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 0x7f8e40033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8e4c0096a0 tx=0x7f8e4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:50.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.708+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e4801d3d0 con 0x7f8e580f52d0 2023-11-16T14:01:50.875 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.873+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8e580fadc0 con 0x7f8e580f52d0 2023-11-16T14:01:50.928 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.926+0000 7f8e46ffd700 1 -- 172.21.15.61:0/1909773640 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8e4801cb10 con 0x7f8e580f52d0 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 msgr2=0x7f8e40033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 0x7f8e40033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8e4c0096a0 tx=0x7f8e4c006b40).stop 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 msgr2=0x7f8e581092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8e480089f0 tx=0x7f8e4800b040).stop 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 shutdown_connections 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8e40031090 0x7f8e40033550 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 --2- 172.21.15.61:0/1909773640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e580f52d0 0x7f8e581092d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 >> 172.21.15.61:0/1909773640 conn(0x7f8e580f0a30 msgr2=0x7f8e580f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 shutdown_connections 2023-11-16T14:01:50.930 INFO:teuthology.orchestra.run.smithi061.stderr:2023-11-16T14:01:50.927+0000 7f8e5759e700 1 -- 172.21.15.61:0/1909773640 wait complete. 2023-11-16T14:01:50.931 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2023-11-16T14:01:51.204 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:50.874377+0000 mon.smithi061 (mon.0) 102 : audit [INF] from='client.? 172.21.15.61:0/1909773640' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T14:01:51.287 INFO:tasks.cephadm:Adding mon.smithi061 on smithi061 2023-11-16T14:01:51.288 INFO:tasks.cephadm:Adding mon.smithi112 on smithi112 2023-11-16T14:01:51.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi061:172.21.15.61=smithi061;smithi112:172.21.15.112=smithi112' 2023-11-16T14:01:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:50.926941+0000 mon.smithi061 (mon.0) 103 : audit [INF] from='client.? 172.21.15.61:0/1909773640' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T14:01:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:50.926982+0000 mon.smithi061 (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-16T14:01:52.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 -- 172.21.15.112:0/513596831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 msgr2=0x7fc93c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 --2- 172.21.15.112:0/513596831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c0f69b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fc930004760 tx=0x7fc930009ad0).stop 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 -- 172.21.15.112:0/513596831 shutdown_connections 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 --2- 172.21.15.112:0/513596831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c0f69b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 -- 172.21.15.112:0/513596831 >> 172.21.15.112:0/513596831 conn(0x7fc93c0f1cd0 msgr2=0x7fc93c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 -- 172.21.15.112:0/513596831 shutdown_connections 2023-11-16T14:01:52.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.390+0000 7fc941527700 1 -- 172.21.15.112:0/513596831 wait complete. 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.391+0000 7fc941527700 1 Processor -- start 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.391+0000 7fc941527700 1 -- start start 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.391+0000 7fc941527700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.391+0000 7fc941527700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c106cb0 con 0x7fc93c0f6590 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.391+0000 7fc93affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:52.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc93affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57920/0 (socket says 172.21.15.112:57920) 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc93affd700 1 -- 172.21.15.112:0/1286152286 learned_addr learned my addr 172.21.15.112:0/1286152286 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc93affd700 1 -- 172.21.15.112:0/1286152286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930009d70 con 0x7fc93c0f6590 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc93affd700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc930004170 tx=0x7fc930009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc93000e040 con 0x7fc93c0f6590 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c106eb0 con 0x7fc93c0f6590 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc930008ef0 con 0x7fc93c0f6590 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc930014c90 con 0x7fc93c0f6590 2023-11-16T14:01:52.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.392+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c107340 con 0x7fc93c0f6590 2023-11-16T14:01:52.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.393+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c045bb0 con 0x7fc93c0f6590 2023-11-16T14:01:52.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.393+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc930014420 con 0x7fc93c0f6590 2023-11-16T14:01:52.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.393+0000 7fc9237fe700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 0x7fc924033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:52.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.393+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc930042220 con 0x7fc93c0f6590 2023-11-16T14:01:52.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.393+0000 7fc93a7fc700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 0x7fc924033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:52.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.394+0000 7fc93a7fc700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 0x7fc924033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc92c0096a0 tx=0x7fc92c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:52.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.398+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc930012d00 con 0x7fc93c0f6590 2023-11-16T14:01:52.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.557+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 --> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi061:172.21.15.61=smithi061;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7fc93c0f2f80 con 0x7fc924031090 2023-11-16T14:01:52.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.562+0000 7fc9237fe700 1 -- 172.21.15.112:0/1286152286 <== mgr.14156 v2:172.21.15.61:6800/4132389775 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc93c0f2f80 con 0x7fc924031090 2023-11-16T14:01:52.564 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mon update... 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 msgr2=0x7fc924033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 0x7fc924033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc92c0096a0 tx=0x7fc92c006b40).stop 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 msgr2=0x7fc93c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc930004170 tx=0x7fc930009ad0).stop 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 shutdown_connections 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc924031090 0x7fc924033550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 --2- 172.21.15.112:0/1286152286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc93c0f6590 0x7fc93c1067c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 >> 172.21.15.112:0/1286152286 conn(0x7fc93c0f1cd0 msgr2=0x7fc93c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 shutdown_connections 2023-11-16T14:01:52.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:52.564+0000 7fc941527700 1 -- 172.21.15.112:0/1286152286 wait complete. 2023-11-16T14:01:52.928 DEBUG:teuthology.orchestra.run.smithi112:mon.smithi112> sudo journalctl -f -n 0 -u ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112.service 2023-11-16T14:01:52.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:01:52.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:01:52.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:-- Logs begin at Thu 2023-11-16 13:47:26 UTC. -- 2023-11-16T14:01:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:52.558681+0000 mgr.smithi061.zcjido (mgr.14156) 10 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi061:172.21.15.61=smithi061;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T14:01:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:01:52.560129+0000 mgr.smithi061.zcjido (mgr.14156) 11 : cephadm [INF] Saving service mon spec with placement smithi061:172.21.15.61=smithi061;smithi112:172.21.15.112=smithi112;count:2 2023-11-16T14:01:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:52.562397+0000 mon.smithi061 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:01:54.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- 172.21.15.112:0/2260028295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 msgr2=0x7f9b880f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:54.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 --2- 172.21.15.112:0/2260028295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b880f8fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9b78009fb0 tx=0x7f9b78009b30).stop 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- 172.21.15.112:0/2260028295 shutdown_connections 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 --2- 172.21.15.112:0/2260028295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b880f8fb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- 172.21.15.112:0/2260028295 >> 172.21.15.112:0/2260028295 conn(0x7f9b880f0a30 msgr2=0x7f9b880f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- 172.21.15.112:0/2260028295 shutdown_connections 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- 172.21.15.112:0/2260028295 wait complete. 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 Processor -- start 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.031+0000 7f9b8c874700 1 -- start start 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b8c874700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b8c874700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b88106aa0 con 0x7f9b88060360 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57934/0 (socket says 172.21.15.112:57934) 2023-11-16T14:01:54.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b86d9d700 1 -- 172.21.15.112:0/3129556666 learned_addr learned my addr 172.21.15.112:0/3129556666 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b86d9d700 1 -- 172.21.15.112:0/3129556666 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78005040 con 0x7f9b88060360 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b86d9d700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9b78009f80 tx=0x7f9b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b78009320 con 0x7f9b88060360 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b88106ca0 con 0x7f9b88060360 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9b78013680 con 0x7f9b88060360 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.032+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b7801eda0 con 0x7f9b88060360 2023-11-16T14:01:54.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.033+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b88107130 con 0x7f9b88060360 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.033+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9b78014420 con 0x7f9b88060360 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.033+0000 7f9b6f7fe700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 0x7f9b70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.033+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9b78042360 con 0x7f9b88060360 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.034+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b88045bb0 con 0x7f9b88060360 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.034+0000 7f9b8659c700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 0x7f9b70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:54.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.034+0000 7f9b8659c700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 0x7f9b70033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9b7c0096a0 tx=0x7f9b7c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:54.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.039+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9b78014700 con 0x7f9b88060360 2023-11-16T14:01:54.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.245+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b88107de0 con 0x7f9b88060360 2023-11-16T14:01:54.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.246+0000 7f9b6f7fe700 1 -- 172.21.15.112:0/3129556666 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b7801c320 con 0x7f9b88060360 2023-11-16T14:01:54.248 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:01:54.248 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:01:54.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.247+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 msgr2=0x7f9b70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:54.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.247+0000 7f9b8c874700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 0x7f9b70033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9b7c0096a0 tx=0x7f9b7c006b40).stop 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.247+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 msgr2=0x7f9b881065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.247+0000 7f9b8c874700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9b78009f80 tx=0x7f9b7800b040).stop 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 shutdown_connections 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9b70031090 0x7f9b70033550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 --2- 172.21.15.112:0/3129556666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b88060360 0x7f9b881065b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 >> 172.21.15.112:0/3129556666 conn(0x7f9b880f0a30 msgr2=0x7f9b880f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 shutdown_connections 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:54.248+0000 7f9b8c874700 1 -- 172.21.15.112:0/3129556666 wait complete. 2023-11-16T14:01:54.250 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:01:55.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:54 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:54.246700+0000 mon.smithi061 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.112:0/3129556666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:01:55.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:01:55.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:01:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:54.838882+0000 mgr.smithi061.zcjido (mgr.14156) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:01:56.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- 172.21.15.112:0/174982798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 msgr2=0x7f01e40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 --2- 172.21.15.112:0/174982798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e40f58a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f01d4009fb0 tx=0x7f01d4009b30).stop 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- 172.21.15.112:0/174982798 shutdown_connections 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 --2- 172.21.15.112:0/174982798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e40f58a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- 172.21.15.112:0/174982798 >> 172.21.15.112:0/174982798 conn(0x7f01e40f0be0 msgr2=0x7f01e40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- 172.21.15.112:0/174982798 shutdown_connections 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- 172.21.15.112:0/174982798 wait complete. 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 Processor -- start 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.659+0000 7f01eb49f700 1 -- start start 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01eb49f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01eb49f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e4060320 con 0x7f01e40f5480 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01e923b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:56.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01e923b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45084/0 (socket says 172.21.15.112:45084) 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01e923b700 1 -- 172.21.15.112:0/3614970487 learned_addr learned my addr 172.21.15.112:0/3614970487 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01e923b700 1 -- 172.21.15.112:0/3614970487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4005040 con 0x7f01e40f5480 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.660+0000 7f01e923b700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f01d40089f0 tx=0x7f01d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.661+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d4009320 con 0x7f01e40f5480 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.661+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e4060520 con 0x7f01e40f5480 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.661+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f01d4013ec0 con 0x7f01e40f5480 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.661+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d4014d30 con 0x7f01e40f5480 2023-11-16T14:01:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.661+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e4109ba0 con 0x7f01e40f5480 2023-11-16T14:01:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f01d4014420 con 0x7f01e40f5480 2023-11-16T14:01:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e4045bb0 con 0x7f01e40f5480 2023-11-16T14:01:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01d9ffb700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 0x7f01d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f01d40423e0 con 0x7f01e40f5480 2023-11-16T14:01:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01e8a3a700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 0x7f01d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.662+0000 7f01e8a3a700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 0x7f01d0033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f01e00096a0 tx=0x7f01e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.667+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f01d401d7f0 con 0x7f01e40f5480 2023-11-16T14:01:56.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.876+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e40faf70 con 0x7f01e40f5480 2023-11-16T14:01:56.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.876+0000 7f01d9ffb700 1 -- 172.21.15.112:0/3614970487 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01d401d7f0 con 0x7f01e40f5480 2023-11-16T14:01:56.878 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:01:56.879 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:01:56.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 msgr2=0x7f01d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 0x7f01d0033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f01e00096a0 tx=0x7f01e0006b40).stop 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 msgr2=0x7f01e41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f01d40089f0 tx=0x7f01d400b040).stop 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 shutdown_connections 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01d0031090 0x7f01d0033550 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 --2- 172.21.15.112:0/3614970487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01e40f5480 0x7f01e41094f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 >> 172.21.15.112:0/3614970487 conn(0x7f01e40f0be0 msgr2=0x7f01e40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.878+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 shutdown_connections 2023-11-16T14:01:56.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:56.879+0000 7f01eb49f700 1 -- 172.21.15.112:0/3614970487 wait complete. 2023-11-16T14:01:56.882 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:01:58.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:01:58.240 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:01:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:56.839237+0000 mgr.smithi061.zcjido (mgr.14156) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:01:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:56.876992+0000 mon.smithi061 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.112:0/3614970487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 -- 172.21.15.112:0/2509675213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 msgr2=0x7f95400f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 --2- 172.21.15.112:0/2509675213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95400f69b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f9530004760 tx=0x7f9530009ad0).stop 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 -- 172.21.15.112:0/2509675213 shutdown_connections 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 --2- 172.21.15.112:0/2509675213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95400f69b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 -- 172.21.15.112:0/2509675213 >> 172.21.15.112:0/2509675213 conn(0x7f95400f1cd0 msgr2=0x7f95400f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 -- 172.21.15.112:0/2509675213 shutdown_connections 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.312+0000 7f95476e3700 1 -- 172.21.15.112:0/2509675213 wait complete. 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f95476e3700 1 Processor -- start 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f95476e3700 1 -- start start 2023-11-16T14:01:59.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f95476e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f95476e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9540106cb0 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f954547f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f954547f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45114/0 (socket says 172.21.15.112:45114) 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f954547f700 1 -- 172.21.15.112:0/3558246564 learned_addr learned my addr 172.21.15.112:0/3558246564 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.313+0000 7f954547f700 1 -- 172.21.15.112:0/3558246564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530009d70 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f954547f700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9530004170 tx=0x7f9530009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953000e040 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540106eb0 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9530008ef0 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9530014c90 con 0x7f95400f6590 2023-11-16T14:01:59.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.314+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9540107340 con 0x7f95400f6590 2023-11-16T14:01:59.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.315+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f95300283f0 con 0x7f95400f6590 2023-11-16T14:01:59.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.315+0000 7f9535ffb700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 0x7f952c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:01:59.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.315+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9530042300 con 0x7f95400f6590 2023-11-16T14:01:59.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.315+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540045bb0 con 0x7f95400f6590 2023-11-16T14:01:59.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.315+0000 7f9544c7e700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 0x7f952c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:01:59.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.316+0000 7f9544c7e700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 0x7f952c033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f953c0096a0 tx=0x7f953c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:01:59.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.320+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9530014420 con 0x7f95400f6590 2023-11-16T14:01:59.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.526+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9540107fc0 con 0x7f95400f6590 2023-11-16T14:01:59.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.527+0000 7f9535ffb700 1 -- 172.21.15.112:0/3558246564 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9530012b20 con 0x7f95400f6590 2023-11-16T14:01:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:01:59.530 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:01:59.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 msgr2=0x7f952c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:59.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 0x7f952c033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f953c0096a0 tx=0x7f953c006b40).stop 2023-11-16T14:01:59.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 msgr2=0x7f95401067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9530004170 tx=0x7f9530009ad0).stop 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 shutdown_connections 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f952c031090 0x7f952c033550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 --2- 172.21.15.112:0/3558246564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95400f6590 0x7f95401067c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 >> 172.21.15.112:0/3558246564 conn(0x7f95400f1cd0 msgr2=0x7f95400f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 shutdown_connections 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:01:59.528+0000 7f95476e3700 1 -- 172.21.15.112:0/3558246564 wait complete. 2023-11-16T14:01:59.532 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:01:58.839609+0000 mgr.smithi061.zcjido (mgr.14156) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:01:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:01:59.528348+0000 mon.smithi061 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.112:0/3558246564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:00.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:00.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- 172.21.15.112:0/112691446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 msgr2=0x7fe8f80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 --2- 172.21.15.112:0/112691446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f80f58d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe8ec009fb0 tx=0x7fe8ec009b30).stop 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- 172.21.15.112:0/112691446 shutdown_connections 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 --2- 172.21.15.112:0/112691446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f80f58d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- 172.21.15.112:0/112691446 >> 172.21.15.112:0/112691446 conn(0x7fe8f80f0c10 msgr2=0x7fe8f80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- 172.21.15.112:0/112691446 shutdown_connections 2023-11-16T14:02:02.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- 172.21.15.112:0/112691446 wait complete. 2023-11-16T14:02:02.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 Processor -- start 2023-11-16T14:02:02.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- start start 2023-11-16T14:02:02.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:02.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.044+0000 7fe8febdf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f8106d00 con 0x7fe8f80f54b0 2023-11-16T14:02:02.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8fc97b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8fc97b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45134/0 (socket says 172.21.15.112:45134) 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8fc97b700 1 -- 172.21.15.112:0/3929513377 learned_addr learned my addr 172.21.15.112:0/3929513377 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8fc97b700 1 -- 172.21.15.112:0/3929513377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec005040 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8fc97b700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe8ec0089f0 tx=0x7fe8ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8ec009320 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8106f00 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe8ec013ec0 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.045+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8ec014d30 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.046+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f8107390 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.046+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe8ec014420 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.046+0000 7fe8f57fa700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.046+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8045bb0 con 0x7fe8f80f54b0 2023-11-16T14:02:02.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.047+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8ec0424c0 con 0x7fe8f80f54b0 2023-11-16T14:02:02.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.047+0000 7fe8f7fff700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:02.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.047+0000 7fe8f7fff700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe8e40096a0 tx=0x7fe8e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:02.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.051+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe8ec01d980 con 0x7fe8f80f54b0 2023-11-16T14:02:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:00.839964+0000 mgr.smithi061.zcjido (mgr.14156) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:02.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.260+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8f8108210 con 0x7fe8f80f54b0 2023-11-16T14:02:02.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.260+0000 7fe8f57fa700 1 -- 172.21.15.112:0/3929513377 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8ec01d380 con 0x7fe8f80f54b0 2023-11-16T14:02:02.263 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:02.264 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:02.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 msgr2=0x7fe8e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:02.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe8e40096a0 tx=0x7fe8e4006b40).stop 2023-11-16T14:02:02.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 msgr2=0x7fe8f8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:02.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe8ec0089f0 tx=0x7fe8ec00b040).stop 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 shutdown_connections 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 --2- 172.21.15.112:0/3929513377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f54b0 0x7fe8f8106810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 >> 172.21.15.112:0/3929513377 conn(0x7fe8f80f0c10 msgr2=0x7fe8f805fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 shutdown_connections 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:02.262+0000 7fe8febdf700 1 -- 172.21.15.112:0/3929513377 wait complete. 2023-11-16T14:02:02.266 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:03.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:02 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:02.261964+0000 mon.smithi061 (mon.0) 109 : audit [DBG] from='client.? 172.21.15.112:0/3929513377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:03.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:03.627 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:04.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:02.840231+0000 mgr.smithi061.zcjido (mgr.14156) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 -- 172.21.15.112:0/1682477710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 msgr2=0x7f82fc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 --2- 172.21.15.112:0/1682477710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc0f58a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f82ec009fb0 tx=0x7f82ec009b30).stop 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 -- 172.21.15.112:0/1682477710 shutdown_connections 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 --2- 172.21.15.112:0/1682477710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc0f58a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 -- 172.21.15.112:0/1682477710 >> 172.21.15.112:0/1682477710 conn(0x7f82fc0f0be0 msgr2=0x7f82fc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 -- 172.21.15.112:0/1682477710 shutdown_connections 2023-11-16T14:02:04.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.711+0000 7f8304482700 1 -- 172.21.15.112:0/1682477710 wait complete. 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f8304482700 1 Processor -- start 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f8304482700 1 -- start start 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f8304482700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f8304482700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc060300 con 0x7f82fc0f5480 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f830221e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f830221e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45146/0 (socket says 172.21.15.112:45146) 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f830221e700 1 -- 172.21.15.112:0/2899239896 learned_addr learned my addr 172.21.15.112:0/2899239896 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f830221e700 1 -- 172.21.15.112:0/2899239896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec005040 con 0x7f82fc0f5480 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.712+0000 7f830221e700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f82ec0089f0 tx=0x7f82ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:04.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.713+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec009320 con 0x7f82fc0f5480 2023-11-16T14:02:04.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.713+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc060500 con 0x7f82fc0f5480 2023-11-16T14:02:04.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.713+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f82ec013ec0 con 0x7f82fc0f5480 2023-11-16T14:02:04.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.713+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc109bc0 con 0x7f82fc0f5480 2023-11-16T14:02:04.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.713+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec014d30 con 0x7f82fc0f5480 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f82ec0095f0 con 0x7f82fc0f5480 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc0591e0 con 0x7f82fc0f5480 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f82f2ffd700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 0x7f82e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f82ec042440 con 0x7f82fc0f5480 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f8301a1d700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 0x7f82e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:04.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.714+0000 7f8301a1d700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 0x7f82e8033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f82f80096a0 tx=0x7f82f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:04.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.719+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f82ec01d7f0 con 0x7f82fc0f5480 2023-11-16T14:02:04.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.926+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82fc045bb0 con 0x7f82fc0f5480 2023-11-16T14:02:04.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.927+0000 7f82f2ffd700 1 -- 172.21.15.112:0/2899239896 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82ec01d7f0 con 0x7f82fc0f5480 2023-11-16T14:02:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:04.931 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:04.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 msgr2=0x7f82e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:04.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 0x7f82e8033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f82f80096a0 tx=0x7f82f8006b40).stop 2023-11-16T14:02:04.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 msgr2=0x7f82fc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:04.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f82ec0089f0 tx=0x7f82ec00b040).stop 2023-11-16T14:02:04.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 shutdown_connections 2023-11-16T14:02:04.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f82e8031090 0x7f82e8033550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:04.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 --2- 172.21.15.112:0/2899239896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82fc0f5480 0x7f82fc109510 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:04.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 >> 172.21.15.112:0/2899239896 conn(0x7f82fc0f0be0 msgr2=0x7f82fc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:04.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 shutdown_connections 2023-11-16T14:02:04.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:04.928+0000 7f8304482700 1 -- 172.21.15.112:0/2899239896 wait complete. 2023-11-16T14:02:04.936 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:04.840583+0000 mgr.smithi061.zcjido (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:04.928352+0000 mon.smithi061 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.112:0/2899239896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:06.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:06.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:07.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 -- 172.21.15.112:0/1514997732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 msgr2=0x7f0ed80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:07.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 --2- 172.21.15.112:0/1514997732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed80f58a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0ec8009fb0 tx=0x7f0ec8009b30).stop 2023-11-16T14:02:07.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 -- 172.21.15.112:0/1514997732 shutdown_connections 2023-11-16T14:02:07.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 --2- 172.21.15.112:0/1514997732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed80f58a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 -- 172.21.15.112:0/1514997732 >> 172.21.15.112:0/1514997732 conn(0x7f0ed80f0be0 msgr2=0x7f0ed80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 -- 172.21.15.112:0/1514997732 shutdown_connections 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.483+0000 7f0ee006d700 1 -- 172.21.15.112:0/1514997732 wait complete. 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0ee006d700 1 Processor -- start 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0ee006d700 1 -- start start 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0ee006d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0ee006d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8107300 con 0x7f0ed80f5480 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0edde09700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0edde09700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:59742/0 (socket says 172.21.15.112:59742) 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0edde09700 1 -- 172.21.15.112:0/4035063092 learned_addr learned my addr 172.21.15.112:0/4035063092 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0edde09700 1 -- 172.21.15.112:0/4035063092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec8005040 con 0x7f0ed80f5480 2023-11-16T14:02:07.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.484+0000 7f0edde09700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0ec80089f0 tx=0x7f0ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:07.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ec8009320 con 0x7f0ed80f5480 2023-11-16T14:02:07.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed8060300 con 0x7f0ed80f5480 2023-11-16T14:02:07.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0ec8013ec0 con 0x7f0ed80f5480 2023-11-16T14:02:07.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ec8014d30 con 0x7f0ed80f5480 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed810be00 con 0x7f0ed80f5480 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0ec80095f0 con 0x7f0ed80f5480 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.485+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed8045bb0 con 0x7f0ed80f5480 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.486+0000 7f0ece7fc700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 0x7f0ec4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.486+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0ec80424a0 con 0x7f0ed80f5480 2023-11-16T14:02:07.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.486+0000 7f0edd608700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 0x7f0ec4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:07.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.486+0000 7f0edd608700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 0x7f0ec4033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0ed40096a0 tx=0x7f0ed4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:07.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.491+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0ec801d980 con 0x7f0ed80f5480 2023-11-16T14:02:07.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.697+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ed80faf70 con 0x7f0ed80f5480 2023-11-16T14:02:07.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.698+0000 7f0ece7fc700 1 -- 172.21.15.112:0/4035063092 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ec801d380 con 0x7f0ed80f5480 2023-11-16T14:02:07.701 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:07.701 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 msgr2=0x7f0ec4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 0x7f0ec4033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0ed40096a0 tx=0x7f0ed4006b40).stop 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 msgr2=0x7f0ed810b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0ec80089f0 tx=0x7f0ec800b040).stop 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 shutdown_connections 2023-11-16T14:02:07.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ec4031090 0x7f0ec4033550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:07.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 --2- 172.21.15.112:0/4035063092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ed80f5480 0x7f0ed810b910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:07.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 >> 172.21.15.112:0/4035063092 conn(0x7f0ed80f0be0 msgr2=0x7f0ed80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:07.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 shutdown_connections 2023-11-16T14:02:07.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:07.699+0000 7f0ee006d700 1 -- 172.21.15.112:0/4035063092 wait complete. 2023-11-16T14:02:07.704 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:06.840962+0000 mgr.smithi061.zcjido (mgr.14156) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:07.699233+0000 mon.smithi061 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.112:0/4035063092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:09.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:09.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- 172.21.15.112:0/3943904703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 msgr2=0x7fcca40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 --2- 172.21.15.112:0/3943904703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca40f69b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcc94004760 tx=0x7fcc94009ad0).stop 2023-11-16T14:02:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- 172.21.15.112:0/3943904703 shutdown_connections 2023-11-16T14:02:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 --2- 172.21.15.112:0/3943904703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca40f69b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- 172.21.15.112:0/3943904703 >> 172.21.15.112:0/3943904703 conn(0x7fcca40f1cd0 msgr2=0x7fcca40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- 172.21.15.112:0/3943904703 shutdown_connections 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- 172.21.15.112:0/3943904703 wait complete. 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 Processor -- start 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.166+0000 7fccabf08700 1 -- start start 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fccabf08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fccabf08700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4106cb0 con 0x7fcca40f6590 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcca9ca4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcca9ca4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:59756/0 (socket says 172.21.15.112:59756) 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcca9ca4700 1 -- 172.21.15.112:0/1705962180 learned_addr learned my addr 172.21.15.112:0/1705962180 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcca9ca4700 1 -- 172.21.15.112:0/1705962180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc94009d70 con 0x7fcca40f6590 2023-11-16T14:02:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcca9ca4700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcc94004170 tx=0x7fcc94009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9400e040 con 0x7fcca40f6590 2023-11-16T14:02:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca4106eb0 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcc94008ef0 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc94014c90 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.167+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca4107340 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.168+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcc94014420 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.169+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4045bb0 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.169+0000 7fcc9a7fc700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 0x7fcc90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.169+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcc940422e0 con 0x7fcca40f6590 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.169+0000 7fcca94a3700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 0x7fcc90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.169+0000 7fcca94a3700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 0x7fcc90033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcca00096a0 tx=0x7fcca0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.174+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcc94012d00 con 0x7fcca40f6590 2023-11-16T14:02:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:08.841291+0000 mgr.smithi061.zcjido (mgr.14156) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:10.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.380+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcca41081f0 con 0x7fcca40f6590 2023-11-16T14:02:10.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.381+0000 7fcc9a7fc700 1 -- 172.21.15.112:0/1705962180 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc94012b20 con 0x7fcca40f6590 2023-11-16T14:02:10.384 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:10.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:10.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 msgr2=0x7fcc90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:10.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 0x7fcc90033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcca00096a0 tx=0x7fcca0006b40).stop 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 msgr2=0x7fcca41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcc94004170 tx=0x7fcc94009ad0).stop 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 shutdown_connections 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc90031090 0x7fcc90033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 --2- 172.21.15.112:0/1705962180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcca40f6590 0x7fcca41067c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.382+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 >> 172.21.15.112:0/1705962180 conn(0x7fcca40f1cd0 msgr2=0x7fcca40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.383+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 shutdown_connections 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:10.383+0000 7fccabf08700 1 -- 172.21.15.112:0/1705962180 wait complete. 2023-11-16T14:02:10.386 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:11.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:10 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:10.382543+0000 mon.smithi061 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.112:0/1705962180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:11.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:11.744 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:10.841488+0000 mgr.smithi061.zcjido (mgr.14156) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.835+0000 7f5696d06700 1 -- 172.21.15.112:0/2382247474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 msgr2=0x7f56900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.835+0000 7f5696d06700 1 --2- 172.21.15.112:0/2382247474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56900f58a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5684009fb0 tx=0x7f5684009b30).stop 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- 172.21.15.112:0/2382247474 shutdown_connections 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 --2- 172.21.15.112:0/2382247474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56900f58a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- 172.21.15.112:0/2382247474 >> 172.21.15.112:0/2382247474 conn(0x7f56900f0be0 msgr2=0x7f56900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- 172.21.15.112:0/2382247474 shutdown_connections 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- 172.21.15.112:0/2382247474 wait complete. 2023-11-16T14:02:12.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 Processor -- start 2023-11-16T14:02:12.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- start start 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.836+0000 7f5696d06700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690060320 con 0x7f56900f5480 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.837+0000 7f5694aa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.837+0000 7f5694aa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:59772/0 (socket says 172.21.15.112:59772) 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.837+0000 7f5694aa2700 1 -- 172.21.15.112:0/3497957523 learned_addr learned my addr 172.21.15.112:0/3497957523 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.837+0000 7f5694aa2700 1 -- 172.21.15.112:0/3497957523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f56900f5480 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.837+0000 7f5694aa2700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f56840089f0 tx=0x7f568400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.838+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684009320 con 0x7f56900f5480 2023-11-16T14:02:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.838+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5690060520 con 0x7f56900f5480 2023-11-16T14:02:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.838+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5684013ec0 con 0x7f56900f5480 2023-11-16T14:02:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.838+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684014d30 con 0x7f56900f5480 2023-11-16T14:02:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.838+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690109ba0 con 0x7f56900f5480 2023-11-16T14:02:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f56840095f0 con 0x7f56900f5480 2023-11-16T14:02:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5690045bb0 con 0x7f56900f5480 2023-11-16T14:02:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f568d7fa700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5684042420 con 0x7f56900f5480 2023-11-16T14:02:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f568ffff700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.839+0000 7f568ffff700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 0x7f5680033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f567c0096a0 tx=0x7f567c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:12.844+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f56840263a0 con 0x7f56900f5480 2023-11-16T14:02:13.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.052+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56900faf70 con 0x7f56900f5480 2023-11-16T14:02:13.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.053+0000 7f568d7fa700 1 -- 172.21.15.112:0/3497957523 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f568400fe40 con 0x7f56900f5480 2023-11-16T14:02:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:13.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 msgr2=0x7f5680033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 0x7f5680033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f567c0096a0 tx=0x7f567c006b40).stop 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 msgr2=0x7f56901094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f56840089f0 tx=0x7f568400b040).stop 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 shutdown_connections 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5680031090 0x7f5680033550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 --2- 172.21.15.112:0/3497957523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56900f5480 0x7f56901094f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 >> 172.21.15.112:0/3497957523 conn(0x7f56900f0be0 msgr2=0x7f56900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:13.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 shutdown_connections 2023-11-16T14:02:13.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:13.055+0000 7f5696d06700 1 -- 172.21.15.112:0/3497957523 wait complete. 2023-11-16T14:02:13.059 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:12.841718+0000 mgr.smithi061.zcjido (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:13.054802+0000 mon.smithi061 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.112:0/3497957523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:14.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:14.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:15.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 -- 172.21.15.112:0/2141423439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 msgr2=0x7fc1e80f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/2141423439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e80f8fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc1dc009fb0 tx=0x7fc1dc009b30).stop 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 -- 172.21.15.112:0/2141423439 shutdown_connections 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/2141423439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e80f8fb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 -- 172.21.15.112:0/2141423439 >> 172.21.15.112:0/2141423439 conn(0x7fc1e80f0a30 msgr2=0x7fc1e80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 -- 172.21.15.112:0/2141423439 shutdown_connections 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 -- 172.21.15.112:0/2141423439 wait complete. 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.527+0000 7fc1ecd85700 1 Processor -- start 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1ecd85700 1 -- start start 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1ecd85700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:15.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1ecd85700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8106aa0 con 0x7fc1e8060360 2023-11-16T14:02:15.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:15.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51008/0 (socket says 172.21.15.112:51008) 2023-11-16T14:02:15.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1e77fe700 1 -- 172.21.15.112:0/102991490 learned_addr learned my addr 172.21.15.112:0/102991490 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1e77fe700 1 -- 172.21.15.112:0/102991490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc005040 con 0x7fc1e8060360 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1e77fe700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc1dc009f80 tx=0x7fc1dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.528+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1dc009320 con 0x7fc1e8060360 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.529+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e8106ca0 con 0x7fc1e8060360 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.529+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc1dc013680 con 0x7fc1e8060360 2023-11-16T14:02:15.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.529+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1dc01eda0 con 0x7fc1e8060360 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.529+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e8107130 con 0x7fc1e8060360 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.529+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc1dc01e420 con 0x7fc1e8060360 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.530+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e8045bb0 con 0x7fc1e8060360 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.530+0000 7fc1cffff700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 0x7fc1d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.530+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1dc042360 con 0x7fc1e8060360 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.530+0000 7fc1e6ffd700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 0x7fc1d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:15.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.530+0000 7fc1e6ffd700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 0x7fc1d0033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc1d80096a0 tx=0x7fc1d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:15.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.535+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc1dc014cc0 con 0x7fc1e8060360 2023-11-16T14:02:15.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.742+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1e8107f80 con 0x7fc1e8060360 2023-11-16T14:02:15.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.743+0000 7fc1cffff700 1 -- 172.21.15.112:0/102991490 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1dc01cb10 con 0x7fc1e8060360 2023-11-16T14:02:15.746 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:15.746 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 msgr2=0x7fc1d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 0x7fc1d0033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc1d80096a0 tx=0x7fc1d8006b40).stop 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 msgr2=0x7fc1e81065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc1dc009f80 tx=0x7fc1dc00b040).stop 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 shutdown_connections 2023-11-16T14:02:15.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc1d0031090 0x7fc1d0033550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 --2- 172.21.15.112:0/102991490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1e8060360 0x7fc1e81065b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 >> 172.21.15.112:0/102991490 conn(0x7fc1e80f0a30 msgr2=0x7fc1e80f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 shutdown_connections 2023-11-16T14:02:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:15.744+0000 7fc1ecd85700 1 -- 172.21.15.112:0/102991490 wait complete. 2023-11-16T14:02:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:14.841971+0000 mgr.smithi061.zcjido (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:15.744270+0000 mon.smithi061 (mon.0) 114 : audit [DBG] from='client.? 172.21.15.112:0/102991490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:17.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:17.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- 172.21.15.112:0/1984631545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 msgr2=0x7fe6a80f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 --2- 172.21.15.112:0/1984631545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a80f6800 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe6a4009fb0 tx=0x7fe6a4009b30).stop 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- 172.21.15.112:0/1984631545 shutdown_connections 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 --2- 172.21.15.112:0/1984631545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a80f6800 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- 172.21.15.112:0/1984631545 >> 172.21.15.112:0/1984631545 conn(0x7fe6a80f1b40 msgr2=0x7fe6a80f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- 172.21.15.112:0/1984631545 shutdown_connections 2023-11-16T14:02:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- 172.21.15.112:0/1984631545 wait complete. 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 Processor -- start 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- start start 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.198+0000 7fe6af478700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8069f00 con 0x7fe6a80f63e0 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6ae476700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6ae476700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51020/0 (socket says 172.21.15.112:51020) 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6ae476700 1 -- 172.21.15.112:0/3193181365 learned_addr learned my addr 172.21.15.112:0/3193181365 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6ae476700 1 -- 172.21.15.112:0/3193181365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4005040 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6ae476700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe6a4000c00 tx=0x7fe6a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a40095f0 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a806a0a0 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe6a4007e50 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a4014e70 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.199+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a806a530 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.200+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe6a4025070 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.200+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a80591e0 con 0x7fe6a80f63e0 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.200+0000 7fe69effd700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 0x7fe694033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.200+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe6a4041e20 con 0x7fe6a80f63e0 2023-11-16T14:02:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.201+0000 7fe6adc75700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 0x7fe694033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.201+0000 7fe6adc75700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 0x7fe694033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe6980096a0 tx=0x7fe698006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:18.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.205+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe6a400f300 con 0x7fe6a80f63e0 2023-11-16T14:02:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:16.842251+0000 mgr.smithi061.zcjido (mgr.14156) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:18.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.415+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6a8045bb0 con 0x7fe6a80f63e0 2023-11-16T14:02:18.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.416+0000 7fe69effd700 1 -- 172.21.15.112:0/3193181365 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6a400f300 con 0x7fe6a80f63e0 2023-11-16T14:02:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:18.419 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 msgr2=0x7fe694033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 0x7fe694033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe6980096a0 tx=0x7fe698006b40).stop 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 msgr2=0x7fe6a806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe6a4000c00 tx=0x7fe6a400b040).stop 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 shutdown_connections 2023-11-16T14:02:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe694031090 0x7fe694033550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 --2- 172.21.15.112:0/3193181365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a80f63e0 0x7fe6a806ba80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 >> 172.21.15.112:0/3193181365 conn(0x7fe6a80f1b40 msgr2=0x7fe6a80f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 shutdown_connections 2023-11-16T14:02:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:18.417+0000 7fe6af478700 1 -- 172.21.15.112:0/3193181365 wait complete. 2023-11-16T14:02:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:19.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:18.417324+0000 mon.smithi061 (mon.0) 115 : audit [DBG] from='client.? 172.21.15.112:0/3193181365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:19.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:19.789 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:18.842521+0000 mgr.smithi061.zcjido (mgr.14156) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.851+0000 7ff9fb133700 1 -- 172.21.15.112:0/3143248749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 msgr2=0x7ff9f40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.851+0000 7ff9fb133700 1 --2- 172.21.15.112:0/3143248749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f40f56f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff9e4009fb0 tx=0x7ff9e4009b30).stop 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.851+0000 7ff9fb133700 1 -- 172.21.15.112:0/3143248749 shutdown_connections 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 --2- 172.21.15.112:0/3143248749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f40f56f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 -- 172.21.15.112:0/3143248749 >> 172.21.15.112:0/3143248749 conn(0x7ff9f40f0a30 msgr2=0x7ff9f40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 -- 172.21.15.112:0/3143248749 shutdown_connections 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 -- 172.21.15.112:0/3143248749 wait complete. 2023-11-16T14:02:20.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 Processor -- start 2023-11-16T14:02:20.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 -- start start 2023-11-16T14:02:20.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:20.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.852+0000 7ff9fb133700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4060320 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fa131700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fa131700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51024/0 (socket says 172.21.15.112:51024) 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fa131700 1 -- 172.21.15.112:0/940689911 learned_addr learned my addr 172.21.15.112:0/940689911 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fa131700 1 -- 172.21.15.112:0/940689911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4005040 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fa131700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff9e40089f0 tx=0x7ff9e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e4009320 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f4060520 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff9e4013ec0 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e4014d30 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.853+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f41099c0 con 0x7ff9f40f52d0 2023-11-16T14:02:20.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.854+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff9e4014420 con 0x7ff9f40f52d0 2023-11-16T14:02:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.854+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f4045bb0 con 0x7ff9f40f52d0 2023-11-16T14:02:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.854+0000 7ff9eaffd700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 0x7ff9e00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.854+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff9e40424c0 con 0x7ff9f40f52d0 2023-11-16T14:02:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.855+0000 7ff9f9930700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 0x7ff9e00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.855+0000 7ff9f9930700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 0x7ff9e00335a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff9f00096a0 tx=0x7ff9f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:20.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:20.861+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff9e40268c0 con 0x7ff9f40f52d0 2023-11-16T14:02:21.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.066+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9f410a1e0 con 0x7ff9f40f52d0 2023-11-16T14:02:21.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.068+0000 7ff9eaffd700 1 -- 172.21.15.112:0/940689911 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9e401d380 con 0x7ff9f40f52d0 2023-11-16T14:02:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:21.070 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 msgr2=0x7ff9e00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 0x7ff9e00335a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff9f00096a0 tx=0x7ff9f0006b40).stop 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 msgr2=0x7ff9f4109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff9e40089f0 tx=0x7ff9e400b040).stop 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 shutdown_connections 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9e00310e0 0x7ff9e00335a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 --2- 172.21.15.112:0/940689911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9f40f52d0 0x7ff9f4109310 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 >> 172.21.15.112:0/940689911 conn(0x7ff9f40f0a30 msgr2=0x7ff9f40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 shutdown_connections 2023-11-16T14:02:21.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:21.070+0000 7ff9fb133700 1 -- 172.21.15.112:0/940689911 wait complete. 2023-11-16T14:02:21.073 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:20.842869+0000 mgr.smithi061.zcjido (mgr.14156) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:21.068528+0000 mon.smithi061 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.112:0/940689911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:22.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:22.444 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:23.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 -- 172.21.15.112:0/984949403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 msgr2=0x7f353c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 --2- 172.21.15.112:0/984949403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c0f69b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f352c009fb0 tx=0x7f352c009b30).stop 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 -- 172.21.15.112:0/984949403 shutdown_connections 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 --2- 172.21.15.112:0/984949403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c0f69b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 -- 172.21.15.112:0/984949403 >> 172.21.15.112:0/984949403 conn(0x7f353c0f1cf0 msgr2=0x7f353c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 -- 172.21.15.112:0/984949403 shutdown_connections 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.524+0000 7f3540d2d700 1 -- 172.21.15.112:0/984949403 wait complete. 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f3540d2d700 1 Processor -- start 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f3540d2d700 1 -- start start 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f3540d2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:23.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f3540d2d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c106c70 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f353a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f353a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51048/0 (socket says 172.21.15.112:51048) 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f353a59c700 1 -- 172.21.15.112:0/1525536687 learned_addr learned my addr 172.21.15.112:0/1525536687 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.525+0000 7f353a59c700 1 -- 172.21.15.112:0/1525536687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c005040 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f353a59c700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f352c000c00 tx=0x7f352c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f352c0095f0 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c106e70 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f352c007e50 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f352c014e70 con 0x7f353c0f6590 2023-11-16T14:02:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.526+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c107300 con 0x7f353c0f6590 2023-11-16T14:02:23.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f352c025070 con 0x7f353c0f6590 2023-11-16T14:02:23.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353c045bb0 con 0x7f353c0f6590 2023-11-16T14:02:23.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f352affd700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 0x7f3524033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:23.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f352c041da0 con 0x7f353c0f6590 2023-11-16T14:02:23.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f3539d9b700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 0x7f3524033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:23.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.527+0000 7f3539d9b700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 0x7f3524033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f35300096a0 tx=0x7f3530006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:23.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.532+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f352c0146b0 con 0x7f353c0f6590 2023-11-16T14:02:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.742+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f353c108110 con 0x7f353c0f6590 2023-11-16T14:02:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.742+0000 7f352affd700 1 -- 172.21.15.112:0/1525536687 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f352c00f6b0 con 0x7f353c0f6590 2023-11-16T14:02:23.745 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 msgr2=0x7f3524033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 0x7f3524033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f35300096a0 tx=0x7f3530006b40).stop 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 msgr2=0x7f353c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f352c000c00 tx=0x7f352c00b040).stop 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 shutdown_connections 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3524031090 0x7f3524033550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 --2- 172.21.15.112:0/1525536687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f353c0f6590 0x7f353c106780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 >> 172.21.15.112:0/1525536687 conn(0x7f353c0f1cf0 msgr2=0x7f353c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 shutdown_connections 2023-11-16T14:02:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:23.744+0000 7f3540d2d700 1 -- 172.21.15.112:0/1525536687 wait complete. 2023-11-16T14:02:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:22.843258+0000 mgr.smithi061.zcjido (mgr.14156) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:25.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:23.743011+0000 mon.smithi061 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.112:0/1525536687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:25.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:25.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:26.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:24.843531+0000 mgr.smithi061.zcjido (mgr.14156) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:26.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- 172.21.15.112:0/2668774953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a00f52d0 msgr2=0x7f53a00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:26.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 --2- 172.21.15.112:0/2668774953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a00f52d0 0x7f53a00f56f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5390009960 tx=0x7f5390004d10).stop 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- 172.21.15.112:0/2668774953 shutdown_connections 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 --2- 172.21.15.112:0/2668774953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a00f52d0 0x7f53a00f56f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- 172.21.15.112:0/2668774953 >> 172.21.15.112:0/2668774953 conn(0x7f53a00f0a30 msgr2=0x7f53a00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- 172.21.15.112:0/2668774953 shutdown_connections 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- 172.21.15.112:0/2668774953 wait complete. 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 Processor -- start 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.157+0000 7f53a49eb700 1 -- start start 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f53a49eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f53a49eb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a0105f50 con 0x7f53a0105640 2023-11-16T14:02:26.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55802/0 (socket says 172.21.15.112:55802) 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f539f7fe700 1 -- 172.21.15.112:0/829573022 learned_addr learned my addr 172.21.15.112:0/829573022 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f539f7fe700 1 -- 172.21.15.112:0/829573022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390009640 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.158+0000 7f539f7fe700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f539000b630 tx=0x7f5390004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5390017070 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a0106150 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5390018040 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5390013db0 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a0106ca0 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f539001d460 con 0x7f53a0105640 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 0x7f5388033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:26.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.159+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f53900426b0 con 0x7f53a0105640 2023-11-16T14:02:26.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.160+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a00591e0 con 0x7f53a0105640 2023-11-16T14:02:26.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.160+0000 7f539effd700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 0x7f5388033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:26.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.160+0000 7f539effd700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 0x7f5388033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f53940096a0 tx=0x7f5394006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:26.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.164+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f539000b7e0 con 0x7f53a0105640 2023-11-16T14:02:26.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.375+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53a0107330 con 0x7f53a0105640 2023-11-16T14:02:26.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.375+0000 7f5387fff700 1 -- 172.21.15.112:0/829573022 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5390011320 con 0x7f53a0105640 2023-11-16T14:02:26.377 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:26.378 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:26.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 msgr2=0x7f5388033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:26.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 0x7f5388033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f53940096a0 tx=0x7f5394006b40).stop 2023-11-16T14:02:26.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 msgr2=0x7f53a0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:26.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f539000b630 tx=0x7f5390004d10).stop 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 shutdown_connections 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5388031090 0x7f5388033550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 --2- 172.21.15.112:0/829573022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53a0105640 0x7f53a0105a60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 >> 172.21.15.112:0/829573022 conn(0x7f53a00f0a30 msgr2=0x7f53a00f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 shutdown_connections 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:26.377+0000 7f53a49eb700 1 -- 172.21.15.112:0/829573022 wait complete. 2023-11-16T14:02:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:27.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:26.376092+0000 mon.smithi061 (mon.0) 118 : audit [DBG] from='client.? 172.21.15.112:0/829573022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:27.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:27.746 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:28.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:26.843893+0000 mgr.smithi061.zcjido (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 -- 172.21.15.112:0/2018721505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 msgr2=0x7fbd880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 --2- 172.21.15.112:0/2018721505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd880f69b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbd78004760 tx=0x7fbd78009ad0).stop 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 -- 172.21.15.112:0/2018721505 shutdown_connections 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 --2- 172.21.15.112:0/2018721505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd880f69b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 -- 172.21.15.112:0/2018721505 >> 172.21.15.112:0/2018721505 conn(0x7fbd880f1cd0 msgr2=0x7fbd880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 -- 172.21.15.112:0/2018721505 shutdown_connections 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.823+0000 7fbd8faec700 1 -- 172.21.15.112:0/2018721505 wait complete. 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8faec700 1 Processor -- start 2023-11-16T14:02:28.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8faec700 1 -- start start 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8faec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8faec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88106cb0 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8d888700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8d888700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55830/0 (socket says 172.21.15.112:55830) 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8d888700 1 -- 172.21.15.112:0/3546599657 learned_addr learned my addr 172.21.15.112:0/3546599657 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8d888700 1 -- 172.21.15.112:0/3546599657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78009d70 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd8d888700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbd78004170 tx=0x7fbd78009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.824+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7800e040 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88106eb0 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd78008ef0 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd78014c90 con 0x7fbd880f6590 2023-11-16T14:02:28.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88107340 con 0x7fbd880f6590 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbd78014420 con 0x7fbd880f6590 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd7e7fc700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.825+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88045bb0 con 0x7fbd880f6590 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.826+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd780422b0 con 0x7fbd880f6590 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.826+0000 7fbd8d087700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:28.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.826+0000 7fbd8d087700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 0x7fbd74033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:28.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:28.830+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd78026300 con 0x7fbd880f6590 2023-11-16T14:02:29.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.045+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd88108140 con 0x7fbd880f6590 2023-11-16T14:02:29.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.047+0000 7fbd7e7fc700 1 -- 172.21.15.112:0/3546599657 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd7802c2d0 con 0x7fbd880f6590 2023-11-16T14:02:29.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:29.049 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 msgr2=0x7fbd74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 0x7fbd74033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).stop 2023-11-16T14:02:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 msgr2=0x7fbd881067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbd78004170 tx=0x7fbd78009ad0).stop 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 shutdown_connections 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 --2- 172.21.15.112:0/3546599657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd880f6590 0x7fbd881067c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 >> 172.21.15.112:0/3546599657 conn(0x7fbd880f1cd0 msgr2=0x7fbd880f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 shutdown_connections 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:29.048+0000 7fbd8faec700 1 -- 172.21.15.112:0/3546599657 wait complete. 2023-11-16T14:02:29.051 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:28.844235+0000 mgr.smithi061.zcjido (mgr.14156) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:29.047094+0000 mon.smithi061 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.112:0/3546599657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:30.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:30.407 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:31.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 -- 172.21.15.112:0/208243792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 msgr2=0x7f7e2c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:31.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/208243792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c0f69b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7e14004760 tx=0x7f7e14009ad0).stop 2023-11-16T14:02:31.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 -- 172.21.15.112:0/208243792 shutdown_connections 2023-11-16T14:02:31.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/208243792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c0f69b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:31.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 -- 172.21.15.112:0/208243792 >> 172.21.15.112:0/208243792 conn(0x7f7e2c0f1cd0 msgr2=0x7f7e2c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:31.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 -- 172.21.15.112:0/208243792 shutdown_connections 2023-11-16T14:02:31.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.582+0000 7f7e30bbc700 1 -- 172.21.15.112:0/208243792 wait complete. 2023-11-16T14:02:31.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e30bbc700 1 Processor -- start 2023-11-16T14:02:31.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e30bbc700 1 -- start start 2023-11-16T14:02:31.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e30bbc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e30bbc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c106cb0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e2a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e2a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55842/0 (socket says 172.21.15.112:55842) 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.583+0000 7f7e2a59c700 1 -- 172.21.15.112:0/2181077928 learned_addr learned my addr 172.21.15.112:0/2181077928 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e2a59c700 1 -- 172.21.15.112:0/2181077928 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14009d70 con 0x7f7e2c0f6590 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e2a59c700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7e14004170 tx=0x7f7e14009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:31.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1400e040 con 0x7f7e2c0f6590 2023-11-16T14:02:31.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c106eb0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7e14008ef0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e14014c90 con 0x7f7e2c0f6590 2023-11-16T14:02:31.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.584+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c107340 con 0x7f7e2c0f6590 2023-11-16T14:02:31.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.585+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7e14014420 con 0x7f7e2c0f6590 2023-11-16T14:02:31.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.585+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c045bb0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.585+0000 7f7e22ffd700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 0x7f7e18033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:31.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.585+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7e140422b0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.585+0000 7f7e29d9b700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 0x7f7e18033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:31.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.586+0000 7f7e29d9b700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 0x7f7e18033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:31.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.590+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7e14026300 con 0x7f7e2c0f6590 2023-11-16T14:02:31.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.800+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e2c1081f0 con 0x7f7e2c0f6590 2023-11-16T14:02:31.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.800+0000 7f7e22ffd700 1 -- 172.21.15.112:0/2181077928 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e14012b20 con 0x7f7e2c0f6590 2023-11-16T14:02:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:31.803 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 msgr2=0x7f7e18033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 0x7f7e18033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).stop 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 msgr2=0x7f7e2c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7e14004170 tx=0x7f7e14009ad0).stop 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 shutdown_connections 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7e18031090 0x7f7e18033550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 --2- 172.21.15.112:0/2181077928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e2c0f6590 0x7f7e2c1067c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 >> 172.21.15.112:0/2181077928 conn(0x7f7e2c0f1cd0 msgr2=0x7f7e2c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 shutdown_connections 2023-11-16T14:02:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:31.802+0000 7f7e30bbc700 1 -- 172.21.15.112:0/2181077928 wait complete. 2023-11-16T14:02:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:30.844476+0000 mgr.smithi061.zcjido (mgr.14156) 30 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:31.800865+0000 mon.smithi061 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.112:0/2181077928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:33.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:33.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:32.844738+0000 mgr.smithi061.zcjido (mgr.14156) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:34.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 -- 172.21.15.112:0/1604773208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 msgr2=0x7fdcf40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:34.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/1604773208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf40f69b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdce4004760 tx=0x7fdce4009ad0).stop 2023-11-16T14:02:34.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 -- 172.21.15.112:0/1604773208 shutdown_connections 2023-11-16T14:02:34.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/1604773208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf40f69b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:34.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 -- 172.21.15.112:0/1604773208 >> 172.21.15.112:0/1604773208 conn(0x7fdcf40f1cd0 msgr2=0x7fdcf40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:34.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 -- 172.21.15.112:0/1604773208 shutdown_connections 2023-11-16T14:02:34.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.229+0000 7fdcfa66b700 1 -- 172.21.15.112:0/1604773208 wait complete. 2023-11-16T14:02:34.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.230+0000 7fdcfa66b700 1 Processor -- start 2023-11-16T14:02:34.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.230+0000 7fdcfa66b700 1 -- start start 2023-11-16T14:02:34.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.230+0000 7fdcfa66b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:34.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.230+0000 7fdcfa66b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf4106cb0 con 0x7fdcf40f6590 2023-11-16T14:02:34.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.230+0000 7fdcf3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55862/0 (socket says 172.21.15.112:55862) 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf3fff700 1 -- 172.21.15.112:0/3369699334 learned_addr learned my addr 172.21.15.112:0/3369699334 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf3fff700 1 -- 172.21.15.112:0/3369699334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009d70 con 0x7fdcf40f6590 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf3fff700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdce4004170 tx=0x7fdce4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce400e040 con 0x7fdcf40f6590 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf4106eb0 con 0x7fdcf40f6590 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdce4008ef0 con 0x7fdcf40f6590 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce4014c90 con 0x7fdcf40f6590 2023-11-16T14:02:34.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.231+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf4107340 con 0x7fdcf40f6590 2023-11-16T14:02:34.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.232+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdce4014420 con 0x7fdcf40f6590 2023-11-16T14:02:34.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.232+0000 7fdcf0ff9700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 0x7fdcdc0335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:34.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.232+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdce40422b0 con 0x7fdcf40f6590 2023-11-16T14:02:34.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.232+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf4045bb0 con 0x7fdcf40f6590 2023-11-16T14:02:34.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.232+0000 7fdcf37fe700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 0x7fdcdc0335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:34.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.233+0000 7fdcf37fe700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 0x7fdcdc0335f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdce80096a0 tx=0x7fdce8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:34.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.237+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdce4012d00 con 0x7fdcf40f6590 2023-11-16T14:02:34.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.445+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcf40fc170 con 0x7fdcf40f6590 2023-11-16T14:02:34.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.446+0000 7fdcf0ff9700 1 -- 172.21.15.112:0/3369699334 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdce4012b20 con 0x7fdcf40f6590 2023-11-16T14:02:34.448 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:34.449 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:34.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 msgr2=0x7fdcdc0335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 0x7fdcdc0335f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdce80096a0 tx=0x7fdce8006b40).stop 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 msgr2=0x7fdcf41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdce4004170 tx=0x7fdce4009ad0).stop 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 shutdown_connections 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcdc031130 0x7fdcdc0335f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:34.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 --2- 172.21.15.112:0/3369699334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcf40f6590 0x7fdcf41067c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 >> 172.21.15.112:0/3369699334 conn(0x7fdcf40f1cd0 msgr2=0x7fdcf40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 shutdown_connections 2023-11-16T14:02:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:34.448+0000 7fdcfa66b700 1 -- 172.21.15.112:0/3369699334 wait complete. 2023-11-16T14:02:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:35.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:34.446507+0000 mon.smithi061 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.112:0/3369699334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:35.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:35.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:34.844963+0000 mgr.smithi061.zcjido (mgr.14156) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:34.978116+0000 mon.smithi061 (mon.0) 122 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:02:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:34.979169+0000 mon.smithi061 (mon.0) 123 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- 172.21.15.112:0/2935204832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 msgr2=0x7feef00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 --2- 172.21.15.112:0/2935204832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef00f58a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7feee4009fb0 tx=0x7feee4009b30).stop 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- 172.21.15.112:0/2935204832 shutdown_connections 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 --2- 172.21.15.112:0/2935204832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef00f58a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- 172.21.15.112:0/2935204832 >> 172.21.15.112:0/2935204832 conn(0x7feef00f0be0 msgr2=0x7feef00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- 172.21.15.112:0/2935204832 shutdown_connections 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- 172.21.15.112:0/2935204832 wait complete. 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 Processor -- start 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.890+0000 7feef6dad700 1 -- start start 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef6dad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef6dad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0060320 con 0x7feef00f5480 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef4b49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef4b49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45746/0 (socket says 172.21.15.112:45746) 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef4b49700 1 -- 172.21.15.112:0/2415379987 learned_addr learned my addr 172.21.15.112:0/2415379987 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef4b49700 1 -- 172.21.15.112:0/2415379987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4005040 con 0x7feef00f5480 2023-11-16T14:02:36.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feef4b49700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feee40089f0 tx=0x7feee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.891+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4009320 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.892+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef0060520 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.892+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7feee4013ec0 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.892+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4014d30 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.892+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef0109ba0 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7feee40095f0 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0045bb0 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feeed7fa700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 0x7feee0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7feee40424a0 con 0x7feef00f5480 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feeeffff700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 0x7feee0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:36.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.893+0000 7feeeffff700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 0x7feee0033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feedc0096a0 tx=0x7feedc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:36.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:36.898+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7feee401d380 con 0x7feef00f5480 2023-11-16T14:02:37.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.107+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feef00faf70 con 0x7feef00f5480 2023-11-16T14:02:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.108+0000 7feeed7fa700 1 -- 172.21.15.112:0/2415379987 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feee401d380 con 0x7feef00f5480 2023-11-16T14:02:37.111 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:37.111 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.109+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 msgr2=0x7feee0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.109+0000 7feef6dad700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 0x7feee0033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feedc0096a0 tx=0x7feedc006b40).stop 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.109+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 msgr2=0x7feef01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.109+0000 7feef6dad700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feee40089f0 tx=0x7feee400b040).stop 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 shutdown_connections 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7feee0031090 0x7feee0033550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 --2- 172.21.15.112:0/2415379987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feef00f5480 0x7feef01094f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 >> 172.21.15.112:0/2415379987 conn(0x7feef00f0be0 msgr2=0x7feef00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 shutdown_connections 2023-11-16T14:02:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:37.110+0000 7feef6dad700 1 -- 172.21.15.112:0/2415379987 wait complete. 2023-11-16T14:02:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:36.845261+0000 mgr.smithi061.zcjido (mgr.14156) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:37.109410+0000 mon.smithi061 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.112:0/2415379987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:38.500 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:38.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:39.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- 172.21.15.112:0/3661894856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 msgr2=0x7f976c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:39.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 --2- 172.21.15.112:0/3661894856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c0f69b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9754004760 tx=0x7f9754009ad0).stop 2023-11-16T14:02:39.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- 172.21.15.112:0/3661894856 shutdown_connections 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 --2- 172.21.15.112:0/3661894856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c0f69b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- 172.21.15.112:0/3661894856 >> 172.21.15.112:0/3661894856 conn(0x7f976c0f1cd0 msgr2=0x7f976c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- 172.21.15.112:0/3661894856 shutdown_connections 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- 172.21.15.112:0/3661894856 wait complete. 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 Processor -- start 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.615+0000 7f9770e7a700 1 -- start start 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f9770e7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:39.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f9770e7a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976c106cb0 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f976a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f976a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45762/0 (socket says 172.21.15.112:45762) 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f976a59c700 1 -- 172.21.15.112:0/2530025597 learned_addr learned my addr 172.21.15.112:0/2530025597 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.616+0000 7f976a59c700 1 -- 172.21.15.112:0/2530025597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754009d70 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f976a59c700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9754004170 tx=0x7f9754009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f975400e040 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976c106eb0 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9754008ef0 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9754014c90 con 0x7f976c0f6590 2023-11-16T14:02:39.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.617+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976c107340 con 0x7f976c0f6590 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.618+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9754014420 con 0x7f976c0f6590 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.618+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976c045bb0 con 0x7f976c0f6590 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.618+0000 7f9762ffd700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 0x7f9758033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.618+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f97540422b0 con 0x7f976c0f6590 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.618+0000 7f9769d9b700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 0x7f9758033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:39.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.619+0000 7f9769d9b700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 0x7f9758033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f975c0096a0 tx=0x7f975c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:39.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.623+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9754026300 con 0x7f976c0f6590 2023-11-16T14:02:39.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.832+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f976c1081f0 con 0x7f976c0f6590 2023-11-16T14:02:39.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.833+0000 7f9762ffd700 1 -- 172.21.15.112:0/2530025597 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9754012b20 con 0x7f976c0f6590 2023-11-16T14:02:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:39.836 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:39.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.834+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 msgr2=0x7f9758033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:39.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.834+0000 7f9770e7a700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 0x7f9758033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f975c0096a0 tx=0x7f975c006b40).stop 2023-11-16T14:02:39.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.834+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 msgr2=0x7f976c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:39.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.834+0000 7f9770e7a700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9754004170 tx=0x7f9754009ad0).stop 2023-11-16T14:02:39.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 shutdown_connections 2023-11-16T14:02:39.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9758031090 0x7f9758033550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:39.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 --2- 172.21.15.112:0/2530025597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f976c0f6590 0x7f976c1067c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:39.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 >> 172.21.15.112:0/2530025597 conn(0x7f976c0f1cd0 msgr2=0x7f976c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:39.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 shutdown_connections 2023-11-16T14:02:39.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:39.835+0000 7f9770e7a700 1 -- 172.21.15.112:0/2530025597 wait complete. 2023-11-16T14:02:39.840 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:38.845509+0000 mgr.smithi061.zcjido (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:39.834595+0000 mon.smithi061 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.112:0/2530025597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:41.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:41.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:40.845871+0000 mgr.smithi061.zcjido (mgr.14156) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 -- 172.21.15.112:0/4253054442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 msgr2=0x7f9c100f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 --2- 172.21.15.112:0/4253054442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c100f58a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f9c04009fb0 tx=0x7f9c04009b30).stop 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 -- 172.21.15.112:0/4253054442 shutdown_connections 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 --2- 172.21.15.112:0/4253054442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c100f58a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 -- 172.21.15.112:0/4253054442 >> 172.21.15.112:0/4253054442 conn(0x7f9c100f0be0 msgr2=0x7f9c100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 -- 172.21.15.112:0/4253054442 shutdown_connections 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.287+0000 7f9c170b7700 1 -- 172.21.15.112:0/4253054442 wait complete. 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c170b7700 1 Processor -- start 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c170b7700 1 -- start start 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c170b7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c170b7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10105be0 con 0x7f9c100f5480 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c14e53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:42.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c14e53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45778/0 (socket says 172.21.15.112:45778) 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c14e53700 1 -- 172.21.15.112:0/2211034512 learned_addr learned my addr 172.21.15.112:0/2211034512 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.288+0000 7f9c14e53700 1 -- 172.21.15.112:0/2211034512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04005040 con 0x7f9c100f5480 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c14e53700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9c040089f0 tx=0x7f9c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c04009320 con 0x7f9c100f5480 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c10105de0 con 0x7f9c100f5480 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9c04013ec0 con 0x7f9c100f5480 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c04014d30 con 0x7f9c100f5480 2023-11-16T14:02:42.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.289+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c10106270 con 0x7f9c100f5480 2023-11-16T14:02:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.290+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9c04014420 con 0x7f9c100f5480 2023-11-16T14:02:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.290+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c10045bb0 con 0x7f9c100f5480 2023-11-16T14:02:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.290+0000 7f9c0d7fa700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 0x7f9c00033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.290+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9c040423e0 con 0x7f9c100f5480 2023-11-16T14:02:42.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.290+0000 7f9c0ffff700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 0x7f9c00033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:42.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.291+0000 7f9c0ffff700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 0x7f9c00033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bfc0096a0 tx=0x7f9bfc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:42.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.295+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9c0401d7f0 con 0x7f9c100f5480 2023-11-16T14:02:42.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.501+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c100faf70 con 0x7f9c100f5480 2023-11-16T14:02:42.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.502+0000 7f9c0d7fa700 1 -- 172.21.15.112:0/2211034512 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c0401d7f0 con 0x7f9c100f5480 2023-11-16T14:02:42.505 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:42.505 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:42.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 msgr2=0x7f9c00033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 0x7f9c00033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bfc0096a0 tx=0x7f9bfc006b40).stop 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 msgr2=0x7f9c101056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9c040089f0 tx=0x7f9c0400b040).stop 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 shutdown_connections 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9c00031090 0x7f9c00033550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 --2- 172.21.15.112:0/2211034512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c100f5480 0x7f9c101056f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 >> 172.21.15.112:0/2211034512 conn(0x7f9c100f0be0 msgr2=0x7f9c100f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 shutdown_connections 2023-11-16T14:02:42.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:42.504+0000 7f9c170b7700 1 -- 172.21.15.112:0/2211034512 wait complete. 2023-11-16T14:02:42.508 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:43.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:42.503588+0000 mon.smithi061 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.112:0/2211034512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:43.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:43.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:44.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:42.846237+0000 mgr.smithi061.zcjido (mgr.14156) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:44.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- 172.21.15.112:0/2935121482 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 msgr2=0x7f66a40f91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 --2- 172.21.15.112:0/2935121482 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a40f91a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f6698009fb0 tx=0x7f6698009b30).stop 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- 172.21.15.112:0/2935121482 shutdown_connections 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 --2- 172.21.15.112:0/2935121482 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a40f91a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- 172.21.15.112:0/2935121482 >> 172.21.15.112:0/2935121482 conn(0x7f66a40f0be0 msgr2=0x7f66a40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- 172.21.15.112:0/2935121482 shutdown_connections 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- 172.21.15.112:0/2935121482 wait complete. 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 Processor -- start 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- start start 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.916+0000 7f66ab255700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4106c90 con 0x7f66a4060320 2023-11-16T14:02:48.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a8ff1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:48.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a8ff1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45796/0 (socket says 172.21.15.112:45796) 2023-11-16T14:02:48.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a8ff1700 1 -- 172.21.15.112:0/1667553005 learned_addr learned my addr 172.21.15.112:0/1667553005 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:48.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a8ff1700 1 -- 172.21.15.112:0/1667553005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698005040 con 0x7f66a4060320 2023-11-16T14:02:48.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a8ff1700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f6698009f80 tx=0x7f669800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:48.951 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:44.846512+0000 mgr.smithi061.zcjido (mgr.14156) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:48.951 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:45.131740+0000 mon.smithi061 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.112:0/1667553005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:48.951 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:46.846860+0000 mgr.smithi061.zcjido (mgr.14156) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:48.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6698009320 con 0x7f66a4060320 2023-11-16T14:02:48.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a4106e90 con 0x7f66a4060320 2023-11-16T14:02:48.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6698013680 con 0x7f66a4060320 2023-11-16T14:02:48.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669801eda0 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.917+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a4107320 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.918+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f669801e420 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.918+0000 7f66a17fa700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 0x7f6694033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.918+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6698042360 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.918+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a4045bb0 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.919+0000 7f66a3fff700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 0x7f6694033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.919+0000 7f66a3fff700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 0x7f6694033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f66900096a0 tx=0x7f6690006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:44.923+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f669801d380 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.130+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66a41080a0 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.130+0000 7f66a17fa700 1 -- 172.21.15.112:0/1667553005 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6698004490 con 0x7f66a4060320 2023-11-16T14:02:48.953 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:48.954 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 msgr2=0x7f6694033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 0x7f6694033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f66900096a0 tx=0x7f6690006b40).stop 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 msgr2=0x7f66a41067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f6698009f80 tx=0x7f669800b040).stop 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 shutdown_connections 2023-11-16T14:02:48.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6694031090 0x7f6694033550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:48.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 --2- 172.21.15.112:0/1667553005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66a4060320 0x7f66a41067a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:48.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 >> 172.21.15.112:0/1667553005 conn(0x7f66a40f0be0 msgr2=0x7f66a40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:48.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 shutdown_connections 2023-11-16T14:02:48.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:45.132+0000 7f66ab255700 1 -- 172.21.15.112:0/1667553005 wait complete. 2023-11-16T14:02:48.981 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:49.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:49.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:48.847232+0000 mgr.smithi061.zcjido (mgr.14156) 39 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 -- 172.21.15.112:0/3728686176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 msgr2=0x7f313c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 --2- 172.21.15.112:0/3728686176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0f58a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f312c009fb0 tx=0x7f312c009b30).stop 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 -- 172.21.15.112:0/3728686176 shutdown_connections 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 --2- 172.21.15.112:0/3728686176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0f58a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 -- 172.21.15.112:0/3728686176 >> 172.21.15.112:0/3728686176 conn(0x7f313c0f0be0 msgr2=0x7f313c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 -- 172.21.15.112:0/3728686176 shutdown_connections 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.030+0000 7f31445c0700 1 -- 172.21.15.112:0/3728686176 wait complete. 2023-11-16T14:02:51.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f31445c0700 1 Processor -- start 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f31445c0700 1 -- start start 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f31445c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f31445c0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0fb8e0 con 0x7f313c0f5480 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f314235c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f314235c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:52860/0 (socket says 172.21.15.112:52860) 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f314235c700 1 -- 172.21.15.112:0/2247484531 learned_addr learned my addr 172.21.15.112:0/2247484531 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.031+0000 7f314235c700 1 -- 172.21.15.112:0/2247484531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c005040 con 0x7f313c0f5480 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f314235c700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f312c000c00 tx=0x7f312c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:51.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f312c0095f0 con 0x7f313c0f5480 2023-11-16T14:02:51.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c0fba80 con 0x7f313c0f5480 2023-11-16T14:02:51.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f312c007e50 con 0x7f313c0f5480 2023-11-16T14:02:51.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f312c014e70 con 0x7f313c0f5480 2023-11-16T14:02:51.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c0fbf10 con 0x7f313c0f5480 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.032+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f312c025070 con 0x7f313c0f5480 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.033+0000 7f3132ffd700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 0x7f3128033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.033+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f312c041d60 con 0x7f313c0f5480 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.033+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c0591e0 con 0x7f313c0f5480 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.033+0000 7f3141b5b700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 0x7f3128033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:51.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.033+0000 7f3141b5b700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 0x7f3128033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f31380096a0 tx=0x7f3138006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:51.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.038+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f312c00f4e0 con 0x7f313c0f5480 2023-11-16T14:02:51.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.249+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f313c045bb0 con 0x7f313c0f5480 2023-11-16T14:02:51.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.250+0000 7f3132ffd700 1 -- 172.21.15.112:0/2247484531 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f312c02d090 con 0x7f313c0f5480 2023-11-16T14:02:51.253 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:51.253 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:51.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.251+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 msgr2=0x7f3128033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.251+0000 7f31445c0700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 0x7f3128033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f31380096a0 tx=0x7f3138006b40).stop 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.251+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 msgr2=0x7f313c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.251+0000 7f31445c0700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f312c000c00 tx=0x7f312c00b040).stop 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 shutdown_connections 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3128031090 0x7f3128033550 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 --2- 172.21.15.112:0/2247484531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f313c0f5480 0x7f313c0fd460 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 >> 172.21.15.112:0/2247484531 conn(0x7f313c0f0be0 msgr2=0x7f313c05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 shutdown_connections 2023-11-16T14:02:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:51.252+0000 7f31445c0700 1 -- 172.21.15.112:0/2247484531 wait complete. 2023-11-16T14:02:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:50.847576+0000 mgr.smithi061.zcjido (mgr.14156) 40 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:51.251515+0000 mon.smithi061 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.112:0/2247484531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:52.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:52.591 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 -- 172.21.15.112:0/3445593111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 msgr2=0x7f06000f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 --2- 172.21.15.112:0/3445593111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f06000f58a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f05f0009fb0 tx=0x7f05f0009b30).stop 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 -- 172.21.15.112:0/3445593111 shutdown_connections 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 --2- 172.21.15.112:0/3445593111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f06000f58a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 -- 172.21.15.112:0/3445593111 >> 172.21.15.112:0/3445593111 conn(0x7f06000f0be0 msgr2=0x7f06000f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 -- 172.21.15.112:0/3445593111 shutdown_connections 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.651+0000 7f060739b700 1 -- 172.21.15.112:0/3445593111 wait complete. 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f060739b700 1 Processor -- start 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f060739b700 1 -- start start 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f060739b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:53.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f060739b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0600060300 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f0605137700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f0605137700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:52876/0 (socket says 172.21.15.112:52876) 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f0605137700 1 -- 172.21.15.112:0/1504219378 learned_addr learned my addr 172.21.15.112:0/1504219378 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.652+0000 7f0605137700 1 -- 172.21.15.112:0/1504219378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0005040 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f0605137700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f05f00089f0 tx=0x7f05f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f0009320 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0600060500 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f05f0013ec0 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f0014d30 con 0x7f06000f5480 2023-11-16T14:02:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.653+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0600109bc0 con 0x7f06000f5480 2023-11-16T14:02:53.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.654+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f05f00095f0 con 0x7f06000f5480 2023-11-16T14:02:53.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.654+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06000591e0 con 0x7f06000f5480 2023-11-16T14:02:53.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.654+0000 7f05f5ffb700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 0x7f05ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:53.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.654+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f05f00424a0 con 0x7f06000f5480 2023-11-16T14:02:53.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.654+0000 7f0604936700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 0x7f05ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:53.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.655+0000 7f0604936700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 0x7f05ec033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f05fc0096a0 tx=0x7f05fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:53.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.659+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f05f001d7f0 con 0x7f06000f5480 2023-11-16T14:02:53.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.867+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0600045bb0 con 0x7f06000f5480 2023-11-16T14:02:53.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.867+0000 7f05f5ffb700 1 -- 172.21.15.112:0/1504219378 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05f001d7f0 con 0x7f06000f5480 2023-11-16T14:02:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:53.871 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:53.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 msgr2=0x7f05ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 0x7f05ec033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f05fc0096a0 tx=0x7f05fc006b40).stop 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 msgr2=0x7f0600109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f05f00089f0 tx=0x7f05f000b040).stop 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 shutdown_connections 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f05ec031090 0x7f05ec033550 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 --2- 172.21.15.112:0/1504219378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06000f5480 0x7f0600109510 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:53.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 >> 172.21.15.112:0/1504219378 conn(0x7f06000f0be0 msgr2=0x7f06000f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:53.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 shutdown_connections 2023-11-16T14:02:53.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:53.869+0000 7f060739b700 1 -- 172.21.15.112:0/1504219378 wait complete. 2023-11-16T14:02:53.874 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:52.847851+0000 mgr.smithi061.zcjido (mgr.14156) 41 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:55.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:54 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:53.868982+0000 mon.smithi061 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.112:0/1504219378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:55.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:55.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:56.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:54.848183+0000 mgr.smithi061.zcjido (mgr.14156) 42 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.367+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1403364622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 msgr2=0x7ffa980f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.367+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1403364622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa980f9160 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffa88009fb0 tx=0x7ffa88009b30).stop 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.367+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1403364622 shutdown_connections 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.367+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1403364622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa980f9160 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.367+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1403364622 >> 172.21.15.112:0/1403364622 conn(0x7ffa980f0be0 msgr2=0x7ffa980f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.368+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1403364622 shutdown_connections 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.368+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1403364622 wait complete. 2023-11-16T14:02:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.368+0000 7ffa9db3a700 1 Processor -- start 2023-11-16T14:02:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.368+0000 7ffa9db3a700 1 -- start start 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa9db3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa9db3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98106c40 con 0x7ffa98060360 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa977fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa977fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:41966/0 (socket says 172.21.15.112:41966) 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa977fe700 1 -- 172.21.15.112:0/1685659797 learned_addr learned my addr 172.21.15.112:0/1685659797 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:56.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa977fe700 1 -- 172.21.15.112:0/1685659797 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa88005040 con 0x7ffa98060360 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.369+0000 7ffa977fe700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ffa88009f80 tx=0x7ffa8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa88009320 con 0x7ffa98060360 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa98106e40 con 0x7ffa98060360 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ffa88013680 con 0x7ffa98060360 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa8801eda0 con 0x7ffa98060360 2023-11-16T14:02:56.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa981072d0 con 0x7ffa98060360 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ffa8801e420 con 0x7ffa98060360 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.370+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa98045bb0 con 0x7ffa98060360 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.371+0000 7ffa7ffff700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 0x7ffa80033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.371+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ffa88042450 con 0x7ffa98060360 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.371+0000 7ffa96ffd700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 0x7ffa80033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:56.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.371+0000 7ffa96ffd700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 0x7ffa80033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ffa8c0096a0 tx=0x7ffa8c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:56.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.375+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ffa8801d380 con 0x7ffa98060360 2023-11-16T14:02:56.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.588+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa98108120 con 0x7ffa98060360 2023-11-16T14:02:56.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.589+0000 7ffa7ffff700 1 -- 172.21.15.112:0/1685659797 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa8801cb10 con 0x7ffa98060360 2023-11-16T14:02:56.592 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:56.592 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:56.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 msgr2=0x7ffa80033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:56.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 0x7ffa80033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ffa8c0096a0 tx=0x7ffa8c006b40).stop 2023-11-16T14:02:56.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 msgr2=0x7ffa98106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:56.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ffa88009f80 tx=0x7ffa8800b040).stop 2023-11-16T14:02:56.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 shutdown_connections 2023-11-16T14:02:56.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffa80031090 0x7ffa80033550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:56.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 --2- 172.21.15.112:0/1685659797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa98060360 0x7ffa98106750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:56.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 >> 172.21.15.112:0/1685659797 conn(0x7ffa980f0be0 msgr2=0x7ffa980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:56.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 shutdown_connections 2023-11-16T14:02:56.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:56.590+0000 7ffa9db3a700 1 -- 172.21.15.112:0/1685659797 wait complete. 2023-11-16T14:02:56.596 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:02:57.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:56 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:56.589910+0000 mon.smithi061 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.112:0/1685659797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:02:57.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:02:57.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:02:58.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:56.848593+0000 mgr.smithi061.zcjido (mgr.14156) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:02:59.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 -- 172.21.15.112:0/2635722975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 msgr2=0x7f7bb80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/2635722975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb80f69b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7bb4004760 tx=0x7f7bb4009ad0).stop 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 -- 172.21.15.112:0/2635722975 shutdown_connections 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/2635722975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb80f69b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 -- 172.21.15.112:0/2635722975 >> 172.21.15.112:0/2635722975 conn(0x7f7bb80f1cd0 msgr2=0x7f7bb80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 -- 172.21.15.112:0/2635722975 shutdown_connections 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.017+0000 7f7bbfe76700 1 -- 172.21.15.112:0/2635722975 wait complete. 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbfe76700 1 Processor -- start 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbfe76700 1 -- start start 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbfe76700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbfe76700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb8106cb0 con 0x7f7bb80f6590 2023-11-16T14:02:59.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbdc12700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbdc12700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:41982/0 (socket says 172.21.15.112:41982) 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbdc12700 1 -- 172.21.15.112:0/398730355 learned_addr learned my addr 172.21.15.112:0/398730355 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbdc12700 1 -- 172.21.15.112:0/398730355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb4009d70 con 0x7f7bb80f6590 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.018+0000 7f7bbdc12700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7bb4013f10 tx=0x7f7bb4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb400e040 con 0x7f7bb80f6590 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb8106eb0 con 0x7f7bb80f6590 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7bb4007df0 con 0x7f7bb80f6590 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb4014d60 con 0x7f7bb80f6590 2023-11-16T14:02:59.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb8107340 con 0x7f7bb80f6590 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.019+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7bb4014520 con 0x7f7bb80f6590 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.020+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb8045bb0 con 0x7f7bb80f6590 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.020+0000 7f7bae7fc700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 0x7f7ba4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.020+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7bb4042580 con 0x7f7bb80f6590 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.020+0000 7f7bbd411700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 0x7f7ba4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:02:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.020+0000 7f7bbd411700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 0x7f7ba4033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7ba80096a0 tx=0x7f7ba8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:02:59.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.025+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7bb4007960 con 0x7f7bb80f6590 2023-11-16T14:02:59.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.233+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bb81081f0 con 0x7f7bb80f6590 2023-11-16T14:02:59.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.234+0000 7f7bae7fc700 1 -- 172.21.15.112:0/398730355 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bb40262f0 con 0x7f7bb80f6590 2023-11-16T14:02:59.237 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:02:59.237 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:02:59.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 msgr2=0x7f7ba4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:59.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 0x7f7ba4033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7ba80096a0 tx=0x7f7ba8006b40).stop 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 msgr2=0x7f7bb81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7bb4013f10 tx=0x7f7bb4009ad0).stop 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 shutdown_connections 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7ba4031090 0x7f7ba4033550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 --2- 172.21.15.112:0/398730355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb80f6590 0x7f7bb81067c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.235+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 >> 172.21.15.112:0/398730355 conn(0x7f7bb80f1cd0 msgr2=0x7f7bb80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.236+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 shutdown_connections 2023-11-16T14:02:59.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:02:59.236+0000 7f7bbfe76700 1 -- 172.21.15.112:0/398730355 wait complete. 2023-11-16T14:02:59.240 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:02:58.848886+0000 mgr.smithi061.zcjido (mgr.14156) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:02:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:02:59.235234+0000 mon.smithi061 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.112:0/398730355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:00.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:00.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:01.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- 172.21.15.112:0/632475041 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 msgr2=0x7f0cf40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:01.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 --2- 172.21.15.112:0/632475041 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf40f58a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0ce8009fb0 tx=0x7f0ce8009b30).stop 2023-11-16T14:03:01.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- 172.21.15.112:0/632475041 shutdown_connections 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 --2- 172.21.15.112:0/632475041 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf40f58a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- 172.21.15.112:0/632475041 >> 172.21.15.112:0/632475041 conn(0x7f0cf40f0be0 msgr2=0x7f0cf40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- 172.21.15.112:0/632475041 shutdown_connections 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- 172.21.15.112:0/632475041 wait complete. 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 Processor -- start 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.638+0000 7f0cfb087700 1 -- start start 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cfb087700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:01.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cfb087700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4060320 con 0x7f0cf40f5480 2023-11-16T14:03:01.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cf8e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:01.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cf8e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42004/0 (socket says 172.21.15.112:42004) 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cf8e23700 1 -- 172.21.15.112:0/3439173423 learned_addr learned my addr 172.21.15.112:0/3439173423 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cf8e23700 1 -- 172.21.15.112:0/3439173423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8005040 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.639+0000 7f0cf8e23700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0ce8000c00 tx=0x7f0ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce80095f0 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf4060520 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0ce8007e50 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce8014e70 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf4109ba0 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.640+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0ce8025070 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.641+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf4045bb0 con 0x7f0cf40f5480 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.641+0000 7f0cf17fa700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 0x7f0ce4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:01.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.641+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0ce8041dc0 con 0x7f0cf40f5480 2023-11-16T14:03:01.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.641+0000 7f0cf3fff700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 0x7f0ce4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:01.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.641+0000 7f0cf3fff700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 0x7f0ce4033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0ce00096a0 tx=0x7f0ce0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:01.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.646+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0ce800f4e0 con 0x7f0cf40f5480 2023-11-16T14:03:01.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.855+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cf40faf70 con 0x7f0cf40f5480 2023-11-16T14:03:01.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.856+0000 7f0cf17fa700 1 -- 172.21.15.112:0/3439173423 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ce800f300 con 0x7f0cf40f5480 2023-11-16T14:03:01.860 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:01.860 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:01.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.857+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 msgr2=0x7f0ce4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.857+0000 7f0cfb087700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 0x7f0ce4033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0ce00096a0 tx=0x7f0ce0006b40).stop 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.857+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 msgr2=0x7f0cf41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.857+0000 7f0cfb087700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0ce8000c00 tx=0x7f0ce800b040).stop 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 shutdown_connections 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0ce4031090 0x7f0ce4033550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 --2- 172.21.15.112:0/3439173423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cf40f5480 0x7f0cf41094f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 >> 172.21.15.112:0/3439173423 conn(0x7f0cf40f0be0 msgr2=0x7f0cf40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 shutdown_connections 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:01.858+0000 7f0cfb087700 1 -- 172.21.15.112:0/3439173423 wait complete. 2023-11-16T14:03:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:00.849234+0000 mgr.smithi061.zcjido (mgr.14156) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:01.856323+0000 mon.smithi061 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.112:0/3439173423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:03.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:03.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:02.849563+0000 mgr.smithi061.zcjido (mgr.14156) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 -- 172.21.15.112:0/89896974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 msgr2=0x7fe2840f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 --2- 172.21.15.112:0/89896974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2840f91f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe278009fb0 tx=0x7fe278009b30).stop 2023-11-16T14:03:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 -- 172.21.15.112:0/89896974 shutdown_connections 2023-11-16T14:03:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 --2- 172.21.15.112:0/89896974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2840f91f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 -- 172.21.15.112:0/89896974 >> 172.21.15.112:0/89896974 conn(0x7fe2840f0c10 msgr2=0x7fe2840f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:04.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 -- 172.21.15.112:0/89896974 shutdown_connections 2023-11-16T14:03:04.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.385+0000 7fe28891f700 1 -- 172.21.15.112:0/89896974 wait complete. 2023-11-16T14:03:04.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28891f700 1 Processor -- start 2023-11-16T14:03:04.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28891f700 1 -- start start 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28891f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28891f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284106c90 con 0x7fe284060300 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42018/0 (socket says 172.21.15.112:42018) 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28259c700 1 -- 172.21.15.112:0/3204943000 learned_addr learned my addr 172.21.15.112:0/3204943000 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28259c700 1 -- 172.21.15.112:0/3204943000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278005040 con 0x7fe284060300 2023-11-16T14:03:04.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.386+0000 7fe28259c700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe278009f80 tx=0x7fe27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:04.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.387+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe278009320 con 0x7fe284060300 2023-11-16T14:03:04.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.387+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe278013680 con 0x7fe284060300 2023-11-16T14:03:04.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.387+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe27801eda0 con 0x7fe284060300 2023-11-16T14:03:04.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.387+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe284106e90 con 0x7fe284060300 2023-11-16T14:03:04.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.387+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe284107320 con 0x7fe284060300 2023-11-16T14:03:04.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.388+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe27801e420 con 0x7fe284060300 2023-11-16T14:03:04.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.388+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe284045bb0 con 0x7fe284060300 2023-11-16T14:03:04.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.388+0000 7fe272ffd700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 0x7fe26c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:04.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.388+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2780424b0 con 0x7fe284060300 2023-11-16T14:03:04.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.388+0000 7fe281d9b700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 0x7fe26c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:04.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.389+0000 7fe281d9b700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 0x7fe26c0335a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe2740096a0 tx=0x7fe274006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:04.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.393+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe27801d380 con 0x7fe284060300 2023-11-16T14:03:04.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.599+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2841081d0 con 0x7fe284060300 2023-11-16T14:03:04.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.600+0000 7fe272ffd700 1 -- 172.21.15.112:0/3204943000 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe27802b2d0 con 0x7fe284060300 2023-11-16T14:03:04.602 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:04.602 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 msgr2=0x7fe26c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 0x7fe26c0335a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe2740096a0 tx=0x7fe274006b40).stop 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 msgr2=0x7fe2841067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe278009f80 tx=0x7fe27800b040).stop 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 shutdown_connections 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe26c0310e0 0x7fe26c0335a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 --2- 172.21.15.112:0/3204943000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe284060300 0x7fe2841067a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:04.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 >> 172.21.15.112:0/3204943000 conn(0x7fe2840f0c10 msgr2=0x7fe2840f16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:04.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 shutdown_connections 2023-11-16T14:03:04.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:04.602+0000 7fe28891f700 1 -- 172.21.15.112:0/3204943000 wait complete. 2023-11-16T14:03:04.605 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:05.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:04 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:04.600594+0000 mon.smithi061 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.112:0/3204943000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:05.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:05.969 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:06.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:04.849889+0000 mgr.smithi061.zcjido (mgr.14156) 47 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:07.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 -- 172.21.15.112:0/1938231506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 msgr2=0x7f63f80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 --2- 172.21.15.112:0/1938231506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f80f58a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f63e0009fb0 tx=0x7f63e0009b30).stop 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 -- 172.21.15.112:0/1938231506 shutdown_connections 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 --2- 172.21.15.112:0/1938231506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f80f58a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 -- 172.21.15.112:0/1938231506 >> 172.21.15.112:0/1938231506 conn(0x7f63f80f0be0 msgr2=0x7f63f80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 -- 172.21.15.112:0/1938231506 shutdown_connections 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 -- 172.21.15.112:0/1938231506 wait complete. 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.037+0000 7f63fde91700 1 Processor -- start 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63fde91700 1 -- start start 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63fde91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63fde91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8060360 con 0x7f63f80f5480 2023-11-16T14:03:07.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63f77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:07.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63f77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54002/0 (socket says 172.21.15.112:54002) 2023-11-16T14:03:07.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63f77fe700 1 -- 172.21.15.112:0/2674937088 learned_addr learned my addr 172.21.15.112:0/2674937088 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63f77fe700 1 -- 172.21.15.112:0/2674937088 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0005040 con 0x7f63f80f5480 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.038+0000 7f63f77fe700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f63e00089f0 tx=0x7f63e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63e0009320 con 0x7f63f80f5480 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f8060560 con 0x7f63f80f5480 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f63e0013ec0 con 0x7f63f80f5480 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63e0014d30 con 0x7f63f80f5480 2023-11-16T14:03:07.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f8109be0 con 0x7f63f80f5480 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.039+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f63e00095f0 con 0x7f63f80f5480 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.040+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8045bb0 con 0x7f63f80f5480 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.040+0000 7f63effff700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 0x7f63e4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.040+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f63e00424a0 con 0x7f63f80f5480 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.040+0000 7f63f6ffd700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 0x7f63e4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:07.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.040+0000 7f63f6ffd700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 0x7f63e4033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63e80096a0 tx=0x7f63e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:07.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.045+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f63e001d7f0 con 0x7f63f80f5480 2023-11-16T14:03:07.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.254+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63f810a400 con 0x7f63f80f5480 2023-11-16T14:03:07.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.255+0000 7f63effff700 1 -- 172.21.15.112:0/2674937088 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63e001d7f0 con 0x7f63f80f5480 2023-11-16T14:03:07.256 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:07.257 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:07.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.256+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 msgr2=0x7f63e4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:07.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.256+0000 7f63fde91700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 0x7f63e4033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63e80096a0 tx=0x7f63e8006b40).stop 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.256+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 msgr2=0x7f63f81094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.256+0000 7f63fde91700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f63e00089f0 tx=0x7f63e000b040).stop 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 shutdown_connections 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63e4031090 0x7f63e4033550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 --2- 172.21.15.112:0/2674937088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63f80f5480 0x7f63f81094b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 >> 172.21.15.112:0/2674937088 conn(0x7f63f80f0be0 msgr2=0x7f63f80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 shutdown_connections 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:07.257+0000 7f63fde91700 1 -- 172.21.15.112:0/2674937088 wait complete. 2023-11-16T14:03:07.259 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:06.850167+0000 mgr.smithi061.zcjido (mgr.14156) 48 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:07.255153+0000 mon.smithi061 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.112:0/2674937088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:08.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:08.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:09.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- 172.21.15.112:0/3662354764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 msgr2=0x7fae300f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 --2- 172.21.15.112:0/3662354764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae300f58d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fae2c009fb0 tx=0x7fae2c009b30).stop 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- 172.21.15.112:0/3662354764 shutdown_connections 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 --2- 172.21.15.112:0/3662354764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae300f58d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- 172.21.15.112:0/3662354764 >> 172.21.15.112:0/3662354764 conn(0x7fae300f0c10 msgr2=0x7fae300f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- 172.21.15.112:0/3662354764 shutdown_connections 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- 172.21.15.112:0/3662354764 wait complete. 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 Processor -- start 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.688+0000 7fae373ad700 1 -- start start 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae373ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:09.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae373ad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae30105b90 con 0x7fae300f54b0 2023-11-16T14:03:09.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae35149700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:09.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae35149700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54022/0 (socket says 172.21.15.112:54022) 2023-11-16T14:03:09.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae35149700 1 -- 172.21.15.112:0/2958096475 learned_addr learned my addr 172.21.15.112:0/2958096475 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:09.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae35149700 1 -- 172.21.15.112:0/2958096475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c005040 con 0x7fae300f54b0 2023-11-16T14:03:09.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.689+0000 7fae35149700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fae2c0089f0 tx=0x7fae2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:09.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.690+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2c009320 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.690+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fae2c013ec0 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.690+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2c014d30 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.690+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae30105d90 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.690+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae30106220 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fae2c0095f0 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae30045bb0 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae25ffb700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 0x7fae1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fae2c0423e0 con 0x7fae300f54b0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae34948700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 0x7fae1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:09.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.691+0000 7fae34948700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 0x7fae1c033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fae200096a0 tx=0x7fae20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:09.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.696+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fae2c01d7f0 con 0x7fae300f54b0 2023-11-16T14:03:09.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.904+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae301070d0 con 0x7fae300f54b0 2023-11-16T14:03:09.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.905+0000 7fae25ffb700 1 -- 172.21.15.112:0/2958096475 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae2c01d7f0 con 0x7fae300f54b0 2023-11-16T14:03:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:09.907 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 msgr2=0x7fae1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 0x7fae1c033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fae200096a0 tx=0x7fae20006b40).stop 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 msgr2=0x7fae301056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fae2c0089f0 tx=0x7fae2c00b040).stop 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 shutdown_connections 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae1c031090 0x7fae1c033550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 --2- 172.21.15.112:0/2958096475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae300f54b0 0x7fae301056a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 >> 172.21.15.112:0/2958096475 conn(0x7fae300f0c10 msgr2=0x7fae300f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:09.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 shutdown_connections 2023-11-16T14:03:09.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:09.907+0000 7fae373ad700 1 -- 172.21.15.112:0/2958096475 wait complete. 2023-11-16T14:03:09.910 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:08.850525+0000 mgr.smithi061.zcjido (mgr.14156) 49 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:11.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:10 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:09.905831+0000 mon.smithi061 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.112:0/2958096475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:11.283 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:11.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:10.850885+0000 mgr.smithi061.zcjido (mgr.14156) 50 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:12.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 -- 172.21.15.112:0/1722312363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 msgr2=0x7fecfc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:12.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 --2- 172.21.15.112:0/1722312363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc0f69b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fecf0004760 tx=0x7fecf0009ad0).stop 2023-11-16T14:03:12.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 -- 172.21.15.112:0/1722312363 shutdown_connections 2023-11-16T14:03:12.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 --2- 172.21.15.112:0/1722312363 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc0f69b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:12.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 -- 172.21.15.112:0/1722312363 >> 172.21.15.112:0/1722312363 conn(0x7fecfc0f1cd0 msgr2=0x7fecfc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:12.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 -- 172.21.15.112:0/1722312363 shutdown_connections 2023-11-16T14:03:12.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.375+0000 7fed01009700 1 -- 172.21.15.112:0/1722312363 wait complete. 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fed01009700 1 Processor -- start 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fed01009700 1 -- start start 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fed01009700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fed01009700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc106cb0 con 0x7fecfc0f6590 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fecfad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fecfad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54032/0 (socket says 172.21.15.112:54032) 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.376+0000 7fecfad9d700 1 -- 172.21.15.112:0/1355114146 learned_addr learned my addr 172.21.15.112:0/1355114146 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fecfad9d700 1 -- 172.21.15.112:0/1355114146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0009d70 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fecfad9d700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fecf0013f10 tx=0x7fecf0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf000e040 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fecf0007df0 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc106eb0 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf0014d60 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.377+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc107340 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.378+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fecf0007960 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.378+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc045bb0 con 0x7fecfc0f6590 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.378+0000 7fece37fe700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 0x7fece4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:12.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.378+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fecf00425a0 con 0x7fecfc0f6590 2023-11-16T14:03:12.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.378+0000 7fecfa59c700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 0x7fece4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:12.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.379+0000 7fecfa59c700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 0x7fece4033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fecec0096a0 tx=0x7fecec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:12.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.383+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fecf0014520 con 0x7fecfc0f6590 2023-11-16T14:03:12.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.594+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecfc1081f0 con 0x7fecfc0f6590 2023-11-16T14:03:12.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.594+0000 7fece37fe700 1 -- 172.21.15.112:0/1355114146 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecf001d320 con 0x7fecfc0f6590 2023-11-16T14:03:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:12.597 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 msgr2=0x7fece4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 0x7fece4033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fecec0096a0 tx=0x7fecec006b40).stop 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 msgr2=0x7fecfc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fecf0013f10 tx=0x7fecf0009ad0).stop 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 shutdown_connections 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fece4031090 0x7fece4033550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 --2- 172.21.15.112:0/1355114146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecfc0f6590 0x7fecfc1067c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 >> 172.21.15.112:0/1355114146 conn(0x7fecfc0f1cd0 msgr2=0x7fecfc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 shutdown_connections 2023-11-16T14:03:12.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:12.596+0000 7fed01009700 1 -- 172.21.15.112:0/1355114146 wait complete. 2023-11-16T14:03:12.599 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:13.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:13.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:12.594822+0000 mon.smithi061 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.112:0/1355114146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:12.851196+0000 mgr.smithi061.zcjido (mgr.14156) 51 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:15.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 -- 172.21.15.112:0/3912455430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 msgr2=0x7f39380f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:15.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 --2- 172.21.15.112:0/3912455430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f39380f67f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f3928009fb0 tx=0x7f3928009b30).stop 2023-11-16T14:03:15.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 -- 172.21.15.112:0/3912455430 shutdown_connections 2023-11-16T14:03:15.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 --2- 172.21.15.112:0/3912455430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f39380f67f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:15.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 -- 172.21.15.112:0/3912455430 >> 172.21.15.112:0/3912455430 conn(0x7f39380f1b10 msgr2=0x7f39380f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 -- 172.21.15.112:0/3912455430 shutdown_connections 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.054+0000 7f393ce12700 1 -- 172.21.15.112:0/3912455430 wait complete. 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.055+0000 7f393ce12700 1 Processor -- start 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.055+0000 7f393ce12700 1 -- start start 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.055+0000 7f393ce12700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.055+0000 7f393ce12700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938106b30 con 0x7f39380f63d0 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f39377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f39377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54054/0 (socket says 172.21.15.112:54054) 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f39377fe700 1 -- 172.21.15.112:0/1581484238 learned_addr learned my addr 172.21.15.112:0/1581484238 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f39377fe700 1 -- 172.21.15.112:0/1581484238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3928005040 con 0x7f39380f63d0 2023-11-16T14:03:15.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f39377fe700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f39280089f0 tx=0x7f392800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:15.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3928009320 con 0x7f39380f63d0 2023-11-16T14:03:15.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3938106d30 con 0x7f39380f63d0 2023-11-16T14:03:15.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3928013ec0 con 0x7f39380f63d0 2023-11-16T14:03:15.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3928014d30 con 0x7f39380f63d0 2023-11-16T14:03:15.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.056+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39381071c0 con 0x7f39380f63d0 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.057+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f39280095f0 con 0x7f39380f63d0 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.057+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938045bb0 con 0x7f39380f63d0 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.057+0000 7f391ffff700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.057+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3928042420 con 0x7f39380f63d0 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.058+0000 7f3936ffd700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:15.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.058+0000 7f3936ffd700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 0x7f3920033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f392c0096a0 tx=0x7f392c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:15.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.062+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f392801d7f0 con 0x7f39380f63d0 2023-11-16T14:03:15.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.273+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3938108010 con 0x7f39380f63d0 2023-11-16T14:03:15.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.274+0000 7f391ffff700 1 -- 172.21.15.112:0/1581484238 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f392800fe40 con 0x7f39380f63d0 2023-11-16T14:03:15.276 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:15.276 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:15.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 msgr2=0x7f3920033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:15.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 0x7f3920033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f392c0096a0 tx=0x7f392c006b40).stop 2023-11-16T14:03:15.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 msgr2=0x7f3938106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:15.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f39280089f0 tx=0x7f392800b040).stop 2023-11-16T14:03:15.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 shutdown_connections 2023-11-16T14:03:15.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:15.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 --2- 172.21.15.112:0/1581484238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39380f63d0 0x7f3938106640 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:15.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 >> 172.21.15.112:0/1581484238 conn(0x7f39380f1b10 msgr2=0x7f39380f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:15.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 shutdown_connections 2023-11-16T14:03:15.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:15.276+0000 7f393ce12700 1 -- 172.21.15.112:0/1581484238 wait complete. 2023-11-16T14:03:15.280 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:14.851548+0000 mgr.smithi061.zcjido (mgr.14156) 52 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:15.274571+0000 mon.smithi061 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.112:0/1581484238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:16.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:16.642 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- 172.21.15.112:0/476010045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 msgr2=0x7f4bcc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 --2- 172.21.15.112:0/476010045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc0f58a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4bbc009fb0 tx=0x7f4bbc009b30).stop 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- 172.21.15.112:0/476010045 shutdown_connections 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 --2- 172.21.15.112:0/476010045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc0f58a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- 172.21.15.112:0/476010045 >> 172.21.15.112:0/476010045 conn(0x7f4bcc0f0be0 msgr2=0x7f4bcc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- 172.21.15.112:0/476010045 shutdown_connections 2023-11-16T14:03:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- 172.21.15.112:0/476010045 wait complete. 2023-11-16T14:03:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 Processor -- start 2023-11-16T14:03:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.719+0000 7f4bd2655700 1 -- start start 2023-11-16T14:03:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bd2655700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bd2655700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc060300 con 0x7f4bcc0f5480 2023-11-16T14:03:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bcbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bcbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:49024/0 (socket says 172.21.15.112:49024) 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bcbfff700 1 -- 172.21.15.112:0/579144695 learned_addr learned my addr 172.21.15.112:0/579144695 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bcbfff700 1 -- 172.21.15.112:0/579144695 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc005040 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.720+0000 7f4bcbfff700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4bbc0089f0 tx=0x7f4bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc009320 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4bbc013ec0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc060500 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc014d30 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc109bc0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.721+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4bbc0095f0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.722+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc0591e0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.722+0000 7f4bc8ff9700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 0x7f4bb40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.722+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4bbc042420 con 0x7f4bcc0f5480 2023-11-16T14:03:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.722+0000 7f4bcb7fe700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 0x7f4bb40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.722+0000 7f4bcb7fe700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 0x7f4bb40335a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4bc00096a0 tx=0x7f4bc0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:17.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.727+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4bbc01d7f0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.935+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bcc045bb0 con 0x7f4bcc0f5480 2023-11-16T14:03:17.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.936+0000 7f4bc8ff9700 1 -- 172.21.15.112:0/579144695 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bbc00fe40 con 0x7f4bcc0f5480 2023-11-16T14:03:17.939 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:17.939 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 msgr2=0x7f4bb40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 0x7f4bb40335a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4bc00096a0 tx=0x7f4bc0006b40).stop 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 msgr2=0x7f4bcc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4bbc0089f0 tx=0x7f4bbc00b040).stop 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 shutdown_connections 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4bb40310e0 0x7f4bb40335a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 --2- 172.21.15.112:0/579144695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bcc0f5480 0x7f4bcc109510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 >> 172.21.15.112:0/579144695 conn(0x7f4bcc0f0be0 msgr2=0x7f4bcc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 shutdown_connections 2023-11-16T14:03:17.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:17.938+0000 7f4bd2655700 1 -- 172.21.15.112:0/579144695 wait complete. 2023-11-16T14:03:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:16.851798+0000 mgr.smithi061.zcjido (mgr.14156) 53 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:19.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:17.937752+0000 mon.smithi061 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.112:0/579144695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:19.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:19.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:18.852139+0000 mgr.smithi061.zcjido (mgr.14156) 54 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 -- 172.21.15.112:0/1794189410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 msgr2=0x7f95f40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 --2- 172.21.15.112:0/1794189410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f40f69b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f95e4004760 tx=0x7f95e4009ad0).stop 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 -- 172.21.15.112:0/1794189410 shutdown_connections 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 --2- 172.21.15.112:0/1794189410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f40f69b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 -- 172.21.15.112:0/1794189410 >> 172.21.15.112:0/1794189410 conn(0x7f95f40f1cd0 msgr2=0x7f95f40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 -- 172.21.15.112:0/1794189410 shutdown_connections 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 -- 172.21.15.112:0/1794189410 wait complete. 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.393+0000 7f95fa378700 1 Processor -- start 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95fa378700 1 -- start start 2023-11-16T14:03:20.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95fa378700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:20.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95fa378700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4106cb0 con 0x7f95f40f6590 2023-11-16T14:03:20.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:20.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:49052/0 (socket says 172.21.15.112:49052) 2023-11-16T14:03:20.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95f3fff700 1 -- 172.21.15.112:0/3052688115 learned_addr learned my addr 172.21.15.112:0/3052688115 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95f3fff700 1 -- 172.21.15.112:0/3052688115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4009d70 con 0x7f95f40f6590 2023-11-16T14:03:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.394+0000 7f95f3fff700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f95e4013f10 tx=0x7f95e4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e400e040 con 0x7f95f40f6590 2023-11-16T14:03:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f4106eb0 con 0x7f95f40f6590 2023-11-16T14:03:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f95e4007df0 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e4014d60 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f4107340 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.395+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f95e4007960 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.396+0000 7f95f0ff9700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 0x7f95dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.396+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f4045bb0 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.396+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f95e40424e0 con 0x7f95f40f6590 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.396+0000 7f95f37fe700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 0x7f95dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:20.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.396+0000 7f95f37fe700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 0x7f95dc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f95e80096a0 tx=0x7f95e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:20.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.401+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f95e4014520 con 0x7f95f40f6590 2023-11-16T14:03:20.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.608+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95f41081c0 con 0x7f95f40f6590 2023-11-16T14:03:20.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.609+0000 7f95f0ff9700 1 -- 172.21.15.112:0/3052688115 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95e403c090 con 0x7f95f40f6590 2023-11-16T14:03:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:20.612 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:20.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 msgr2=0x7f95dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:20.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 0x7f95dc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f95e80096a0 tx=0x7f95e8006b40).stop 2023-11-16T14:03:20.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 msgr2=0x7f95f41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:20.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f95e4013f10 tx=0x7f95e4009ad0).stop 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 shutdown_connections 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f95dc031090 0x7f95dc033550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 --2- 172.21.15.112:0/3052688115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95f40f6590 0x7f95f41067c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 >> 172.21.15.112:0/3052688115 conn(0x7f95f40f1cd0 msgr2=0x7f95f40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 shutdown_connections 2023-11-16T14:03:20.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:20.611+0000 7f95fa378700 1 -- 172.21.15.112:0/3052688115 wait complete. 2023-11-16T14:03:20.616 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:21.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:20 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:20.610679+0000 mon.smithi061 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.112:0/3052688115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:21.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:21.980 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:20.852492+0000 mgr.smithi061.zcjido (mgr.14156) 55 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:23.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.030+0000 7f4ca96d5700 1 -- 172.21.15.112:0/1288477965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 msgr2=0x7f4ca40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.030+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/1288477965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca40f69b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4c94009fb0 tx=0x7f4c94009b30).stop 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- 172.21.15.112:0/1288477965 shutdown_connections 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/1288477965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca40f69b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- 172.21.15.112:0/1288477965 >> 172.21.15.112:0/1288477965 conn(0x7f4ca40f1cf0 msgr2=0x7f4ca40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- 172.21.15.112:0/1288477965 shutdown_connections 2023-11-16T14:03:23.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- 172.21.15.112:0/1288477965 wait complete. 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 Processor -- start 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- start start 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.031+0000 7f4ca96d5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca4106c70 con 0x7f4ca40f6590 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:49068/0 (socket says 172.21.15.112:49068) 2023-11-16T14:03:23.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca2ffd700 1 -- 172.21.15.112:0/277928253 learned_addr learned my addr 172.21.15.112:0/277928253 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca2ffd700 1 -- 172.21.15.112:0/277928253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c94005040 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca2ffd700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c940089f0 tx=0x7f4c9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c94009320 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca4106e70 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4c94013ec0 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c94014d30 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.032+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4107300 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.033+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4c940263a0 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.033+0000 7f4c8b7fe700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 0x7f4c8c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.033+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca4045bb0 con 0x7f4ca40f6590 2023-11-16T14:03:23.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.033+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4c94042450 con 0x7f4ca40f6590 2023-11-16T14:03:23.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.034+0000 7f4ca27fc700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 0x7f4c8c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:23.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.034+0000 7f4ca27fc700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 0x7f4c8c0335a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f4c980096a0 tx=0x7f4c98006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:23.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.038+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4c9401cb10 con 0x7f4ca40f6590 2023-11-16T14:03:23.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.247+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca40fc170 con 0x7f4ca40f6590 2023-11-16T14:03:23.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.247+0000 7f4c8b7fe700 1 -- 172.21.15.112:0/277928253 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c9402b020 con 0x7f4ca40f6590 2023-11-16T14:03:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:23.250 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 msgr2=0x7f4c8c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 0x7f4c8c0335a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f4c980096a0 tx=0x7f4c98006b40).stop 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 msgr2=0x7f4ca4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c940089f0 tx=0x7f4c9400b040).stop 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 shutdown_connections 2023-11-16T14:03:23.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c0310e0 0x7f4c8c0335a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:23.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 --2- 172.21.15.112:0/277928253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40f6590 0x7f4ca4106780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:23.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 >> 172.21.15.112:0/277928253 conn(0x7f4ca40f1cf0 msgr2=0x7f4ca40f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:23.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 shutdown_connections 2023-11-16T14:03:23.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:23.249+0000 7f4ca96d5700 1 -- 172.21.15.112:0/277928253 wait complete. 2023-11-16T14:03:23.253 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:22.852827+0000 mgr.smithi061.zcjido (mgr.14156) 56 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:23.248870+0000 mon.smithi061 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.112:0/277928253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:24.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:24.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 -- 172.21.15.112:0/194947705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 msgr2=0x7f6c800f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 --2- 172.21.15.112:0/194947705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c800f58a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f6c74009fb0 tx=0x7f6c74009b30).stop 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 -- 172.21.15.112:0/194947705 shutdown_connections 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 --2- 172.21.15.112:0/194947705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c800f58a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 -- 172.21.15.112:0/194947705 >> 172.21.15.112:0/194947705 conn(0x7f6c800f0be0 msgr2=0x7f6c800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 -- 172.21.15.112:0/194947705 shutdown_connections 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 -- 172.21.15.112:0/194947705 wait complete. 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.656+0000 7f6c84917700 1 Processor -- start 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c84917700 1 -- start start 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c84917700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c84917700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80106ce0 con 0x7f6c800f5480 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c7e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c7e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34842/0 (socket says 172.21.15.112:34842) 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c7e59c700 1 -- 172.21.15.112:0/1914812533 learned_addr learned my addr 172.21.15.112:0/1914812533 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c7e59c700 1 -- 172.21.15.112:0/1914812533 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74005040 con 0x7f6c800f5480 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c7e59c700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6c740089f0 tx=0x7f6c7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.657+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74009320 con 0x7f6c800f5480 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80106ee0 con 0x7f6c800f5480 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6c74013ec0 con 0x7f6c800f5480 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74014d30 con 0x7f6c800f5480 2023-11-16T14:03:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c80107370 con 0x7f6c800f5480 2023-11-16T14:03:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6c74014420 con 0x7f6c800f5480 2023-11-16T14:03:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c6effd700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 0x7f6c68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6c740423e0 con 0x7f6c800f5480 2023-11-16T14:03:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.658+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80045bb0 con 0x7f6c800f5480 2023-11-16T14:03:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.659+0000 7f6c7dd9b700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 0x7f6c68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:25.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.659+0000 7f6c7dd9b700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 0x7f6c68033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6c700096a0 tx=0x7f6c70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:25.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.663+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6c74014750 con 0x7f6c800f5480 2023-11-16T14:03:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.872+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c800faf70 con 0x7f6c800f5480 2023-11-16T14:03:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.873+0000 7f6c6effd700 1 -- 172.21.15.112:0/1914812533 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c7401d7f0 con 0x7f6c800f5480 2023-11-16T14:03:25.876 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:25.876 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.874+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 msgr2=0x7f6c68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.874+0000 7f6c84917700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 0x7f6c68033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6c700096a0 tx=0x7f6c70006b40).stop 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.874+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 msgr2=0x7f6c801067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.874+0000 7f6c84917700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6c740089f0 tx=0x7f6c7400b040).stop 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 shutdown_connections 2023-11-16T14:03:25.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6c68031090 0x7f6c68033550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:25.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 --2- 172.21.15.112:0/1914812533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c800f5480 0x7f6c801067f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:25.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 >> 172.21.15.112:0/1914812533 conn(0x7f6c800f0be0 msgr2=0x7f6c8005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:25.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 shutdown_connections 2023-11-16T14:03:25.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:25.875+0000 7f6c84917700 1 -- 172.21.15.112:0/1914812533 wait complete. 2023-11-16T14:03:25.879 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:24.853135+0000 mgr.smithi061.zcjido (mgr.14156) 57 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:25.874283+0000 mon.smithi061 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.112:0/1914812533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:27.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:27.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:26.853480+0000 mgr.smithi061.zcjido (mgr.14156) 58 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:28.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1147195648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 msgr2=0x7fd1ec0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:28.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1147195648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec0f69b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd1e0009fb0 tx=0x7fd1e0009b30).stop 2023-11-16T14:03:28.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1147195648 shutdown_connections 2023-11-16T14:03:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1147195648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec0f69b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1147195648 >> 172.21.15.112:0/1147195648 conn(0x7fd1ec0f1cf0 msgr2=0x7fd1ec0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1147195648 shutdown_connections 2023-11-16T14:03:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.297+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1147195648 wait complete. 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1f21cf700 1 Processor -- start 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1f21cf700 1 -- start start 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1f21cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1f21cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec106c70 con 0x7fd1ec0f6590 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1eb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1eb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34868/0 (socket says 172.21.15.112:34868) 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.298+0000 7fd1eb7fe700 1 -- 172.21.15.112:0/1967355691 learned_addr learned my addr 172.21.15.112:0/1967355691 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1eb7fe700 1 -- 172.21.15.112:0/1967355691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0005040 con 0x7fd1ec0f6590 2023-11-16T14:03:28.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1eb7fe700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd1e00089f0 tx=0x7fd1e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e0009320 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec106e70 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd1e0013ec0 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e0014d30 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.299+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec107300 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.300+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd1e00263a0 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.300+0000 7fd1d3fff700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 0x7fd1d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.300+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd1e00423d0 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.300+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec045bb0 con 0x7fd1ec0f6590 2023-11-16T14:03:28.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.300+0000 7fd1eaffd700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 0x7fd1d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:28.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.301+0000 7fd1eaffd700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 0x7fd1d4033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd1dc0096a0 tx=0x7fd1dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:28.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.305+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd1e00266d0 con 0x7fd1ec0f6590 2023-11-16T14:03:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.511+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1ec107f80 con 0x7fd1ec0f6590 2023-11-16T14:03:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.512+0000 7fd1d3fff700 1 -- 172.21.15.112:0/1967355691 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1e002b020 con 0x7fd1ec0f6590 2023-11-16T14:03:28.515 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:28.515 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.513+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 msgr2=0x7fd1d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.513+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 0x7fd1d4033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd1dc0096a0 tx=0x7fd1dc006b40).stop 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.513+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 msgr2=0x7fd1ec106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.513+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd1e00089f0 tx=0x7fd1e000b040).stop 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.513+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 shutdown_connections 2023-11-16T14:03:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.514+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd1d4031090 0x7fd1d4033550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.514+0000 7fd1f21cf700 1 --2- 172.21.15.112:0/1967355691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1ec0f6590 0x7fd1ec106780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.514+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 >> 172.21.15.112:0/1967355691 conn(0x7fd1ec0f1cf0 msgr2=0x7fd1ec0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.514+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 shutdown_connections 2023-11-16T14:03:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:28.514+0000 7fd1f21cf700 1 -- 172.21.15.112:0/1967355691 wait complete. 2023-11-16T14:03:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:29.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:28.513297+0000 mon.smithi061 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.112:0/1967355691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:29.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:29.926 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:28.853838+0000 mgr.smithi061.zcjido (mgr.14156) 59 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:31.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- 172.21.15.112:0/1453186172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 msgr2=0x7fcf800f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:31.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 --2- 172.21.15.112:0/1453186172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf800f8fb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fcf70009fb0 tx=0x7fcf70009b30).stop 2023-11-16T14:03:31.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- 172.21.15.112:0/1453186172 shutdown_connections 2023-11-16T14:03:31.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 --2- 172.21.15.112:0/1453186172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf800f8fb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- 172.21.15.112:0/1453186172 >> 172.21.15.112:0/1453186172 conn(0x7fcf800f0a30 msgr2=0x7fcf800f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- 172.21.15.112:0/1453186172 shutdown_connections 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- 172.21.15.112:0/1453186172 wait complete. 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 Processor -- start 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 -- start start 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.007+0000 7fcf8530b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf8530b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80106aa0 con 0x7fcf80060360 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34880/0 (socket says 172.21.15.112:34880) 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7ffff700 1 -- 172.21.15.112:0/841316296 learned_addr learned my addr 172.21.15.112:0/841316296 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7ffff700 1 -- 172.21.15.112:0/841316296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf70005040 con 0x7fcf80060360 2023-11-16T14:03:31.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7ffff700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcf70009f80 tx=0x7fcf7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:31.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf70009320 con 0x7fcf80060360 2023-11-16T14:03:31.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf80106ca0 con 0x7fcf80060360 2023-11-16T14:03:31.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcf70013680 con 0x7fcf80060360 2023-11-16T14:03:31.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf7001eda0 con 0x7fcf80060360 2023-11-16T14:03:31.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.008+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf80107130 con 0x7fcf80060360 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.009+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcf70014420 con 0x7fcf80060360 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.009+0000 7fcf7cff9700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 0x7fcf680335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.009+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcf700423b0 con 0x7fcf80060360 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.010+0000 7fcf7f7fe700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 0x7fcf680335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.010+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf800591e0 con 0x7fcf80060360 2023-11-16T14:03:31.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.010+0000 7fcf7f7fe700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 0x7fcf680335a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcf740096a0 tx=0x7fcf74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:31.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.014+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcf7001c910 con 0x7fcf80060360 2023-11-16T14:03:31.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.222+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf80045bb0 con 0x7fcf80060360 2023-11-16T14:03:31.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.223+0000 7fcf7cff9700 1 -- 172.21.15.112:0/841316296 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf7001c320 con 0x7fcf80060360 2023-11-16T14:03:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:31.226 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:31.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 msgr2=0x7fcf680335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:31.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 0x7fcf680335a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcf740096a0 tx=0x7fcf74006b40).stop 2023-11-16T14:03:31.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 msgr2=0x7fcf801065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:31.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcf70009f80 tx=0x7fcf7000b040).stop 2023-11-16T14:03:31.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 shutdown_connections 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcf680310e0 0x7fcf680335a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 --2- 172.21.15.112:0/841316296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf80060360 0x7fcf801065b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 >> 172.21.15.112:0/841316296 conn(0x7fcf800f0a30 msgr2=0x7fcf800f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.224+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 shutdown_connections 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:31.225+0000 7fcf8530b700 1 -- 172.21.15.112:0/841316296 wait complete. 2023-11-16T14:03:31.228 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:30.854163+0000 mgr.smithi061.zcjido (mgr.14156) 60 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:31.224391+0000 mon.smithi061 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.112:0/841316296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:32.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:32.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 -- 172.21.15.112:0/27460005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 msgr2=0x7f4da00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 --2- 172.21.15.112:0/27460005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da00f58a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4d9c009fb0 tx=0x7f4d9c009b30).stop 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 -- 172.21.15.112:0/27460005 shutdown_connections 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 --2- 172.21.15.112:0/27460005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da00f58a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 -- 172.21.15.112:0/27460005 >> 172.21.15.112:0/27460005 conn(0x7f4da00f0be0 msgr2=0x7f4da00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:33.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 -- 172.21.15.112:0/27460005 shutdown_connections 2023-11-16T14:03:33.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.613+0000 7f4da7733700 1 -- 172.21.15.112:0/27460005 wait complete. 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da7733700 1 Processor -- start 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da7733700 1 -- start start 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da7733700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da7733700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0105be0 con 0x7f4da00f5480 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da54cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da54cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34890/0 (socket says 172.21.15.112:34890) 2023-11-16T14:03:33.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.614+0000 7f4da54cf700 1 -- 172.21.15.112:0/841535045 learned_addr learned my addr 172.21.15.112:0/841535045 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4da54cf700 1 -- 172.21.15.112:0/841535045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c005040 con 0x7f4da00f5480 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4da54cf700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f4d9c0089f0 tx=0x7f4d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c009320 con 0x7f4da00f5480 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da0105de0 con 0x7f4da00f5480 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4d9c013ec0 con 0x7f4da00f5480 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d9c014d30 con 0x7f4da00f5480 2023-11-16T14:03:33.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.615+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da0106270 con 0x7f4da00f5480 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.616+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4d9c0095f0 con 0x7f4da00f5480 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.616+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da0045bb0 con 0x7f4da00f5480 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.616+0000 7f4d95ffb700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 0x7f4d8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.616+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4d9c0424a0 con 0x7f4da00f5480 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.616+0000 7f4da4cce700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 0x7f4d8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:33.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.617+0000 7f4da4cce700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 0x7f4d8c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4d900096a0 tx=0x7f4d90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:33.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.621+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4d9c01d7f0 con 0x7f4da00f5480 2023-11-16T14:03:33.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.827+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4da00faf70 con 0x7f4da00f5480 2023-11-16T14:03:33.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.828+0000 7f4d95ffb700 1 -- 172.21.15.112:0/841535045 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d9c00fe40 con 0x7f4da00f5480 2023-11-16T14:03:33.831 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:33.831 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:33.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.829+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 msgr2=0x7f4d8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:33.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.829+0000 7f4da7733700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 0x7f4d8c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4d900096a0 tx=0x7f4d90006b40).stop 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.829+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 msgr2=0x7f4da01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.829+0000 7f4da7733700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f4d9c0089f0 tx=0x7f4d9c00b040).stop 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 shutdown_connections 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4d8c031090 0x7f4d8c033550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 --2- 172.21.15.112:0/841535045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4da00f5480 0x7f4da01056f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 >> 172.21.15.112:0/841535045 conn(0x7f4da00f0be0 msgr2=0x7f4da00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 shutdown_connections 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:33.830+0000 7f4da7733700 1 -- 172.21.15.112:0/841535045 wait complete. 2023-11-16T14:03:33.833 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:32.854503+0000 mgr.smithi061.zcjido (mgr.14156) 61 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:35.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:33.829369+0000 mon.smithi061 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.112:0/841535045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:35.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:35.204 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:34.854798+0000 mgr.smithi061.zcjido (mgr.14156) 62 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:34.979135+0000 mon.smithi061 (mon.0) 145 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:03:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:34.979969+0000 mon.smithi061 (mon.0) 146 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:03:36.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 -- 172.21.15.112:0/112451605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 msgr2=0x7f20780f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:36.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 --2- 172.21.15.112:0/112451605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20780f58a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f2068009fb0 tx=0x7f2068009b30).stop 2023-11-16T14:03:36.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 -- 172.21.15.112:0/112451605 shutdown_connections 2023-11-16T14:03:36.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 --2- 172.21.15.112:0/112451605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20780f58a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 -- 172.21.15.112:0/112451605 >> 172.21.15.112:0/112451605 conn(0x7f20780f0be0 msgr2=0x7f20780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 -- 172.21.15.112:0/112451605 shutdown_connections 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.254+0000 7f2080321700 1 -- 172.21.15.112:0/112451605 wait complete. 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f2080321700 1 Processor -- start 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f2080321700 1 -- start start 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f2080321700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f2080321700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2078105be0 con 0x7f20780f5480 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f207e0bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f207e0bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57318/0 (socket says 172.21.15.112:57318) 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.255+0000 7f207e0bd700 1 -- 172.21.15.112:0/3637282567 learned_addr learned my addr 172.21.15.112:0/3637282567 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f207e0bd700 1 -- 172.21.15.112:0/3637282567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2068005040 con 0x7f20780f5480 2023-11-16T14:03:36.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f207e0bd700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f20680089f0 tx=0x7f206800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068009320 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2068013ec0 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2078105de0 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068014d30 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.256+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2078106270 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f20680095f0 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2078045bb0 con 0x7f20780f5480 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f206effd700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 0x7f2064033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:36.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2068042460 con 0x7f20780f5480 2023-11-16T14:03:36.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f207d8bc700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 0x7f2064033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:36.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.257+0000 7f207d8bc700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 0x7f2064033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f20740096a0 tx=0x7f2074006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:36.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.262+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f206801d7f0 con 0x7f20780f5480 2023-11-16T14:03:36.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.473+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20780faf70 con 0x7f20780f5480 2023-11-16T14:03:36.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.475+0000 7f206effd700 1 -- 172.21.15.112:0/3637282567 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f206801d7f0 con 0x7f20780f5480 2023-11-16T14:03:36.477 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:36.477 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:36.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 msgr2=0x7f2064033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:36.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 0x7f2064033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f20740096a0 tx=0x7f2074006b40).stop 2023-11-16T14:03:36.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 msgr2=0x7f20781056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:36.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f20680089f0 tx=0x7f206800b040).stop 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 shutdown_connections 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2064031090 0x7f2064033550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 --2- 172.21.15.112:0/3637282567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5480 0x7f20781056f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 >> 172.21.15.112:0/3637282567 conn(0x7f20780f0be0 msgr2=0x7f20780f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 shutdown_connections 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:36.476+0000 7f2080321700 1 -- 172.21.15.112:0/3637282567 wait complete. 2023-11-16T14:03:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:37.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:36.475380+0000 mon.smithi061 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.112:0/3637282567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:37.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:37.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:38.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:36.855131+0000 mgr.smithi061.zcjido (mgr.14156) 63 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 -- 172.21.15.112:0/2485869301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 msgr2=0x7f32240f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 --2- 172.21.15.112:0/2485869301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32240f69b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3214004760 tx=0x7f3214009ad0).stop 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 -- 172.21.15.112:0/2485869301 shutdown_connections 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 --2- 172.21.15.112:0/2485869301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32240f69b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 -- 172.21.15.112:0/2485869301 >> 172.21.15.112:0/2485869301 conn(0x7f32240f1cd0 msgr2=0x7f32240f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:38.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.924+0000 7f322b847700 1 -- 172.21.15.112:0/2485869301 shutdown_connections 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f322b847700 1 -- 172.21.15.112:0/2485869301 wait complete. 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f322b847700 1 Processor -- start 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f322b847700 1 -- start start 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f322b847700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f322b847700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3224108ea0 con 0x7f32240f6590 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f32295e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f32295e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57342/0 (socket says 172.21.15.112:57342) 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.925+0000 7f32295e3700 1 -- 172.21.15.112:0/3399928555 learned_addr learned my addr 172.21.15.112:0/3399928555 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f32295e3700 1 -- 172.21.15.112:0/3399928555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009d70 con 0x7f32240f6590 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f32295e3700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3214004170 tx=0x7f3214009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321400e040 con 0x7f32240f6590 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3214008ef0 con 0x7f32240f6590 2023-11-16T14:03:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32241090a0 con 0x7f32240f6590 2023-11-16T14:03:38.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3214014c90 con 0x7f32240f6590 2023-11-16T14:03:38.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.926+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3224109530 con 0x7f32240f6590 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.927+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3214014420 con 0x7f32240f6590 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.927+0000 7f3219ffb700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 0x7f32100335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.927+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224045bb0 con 0x7f32240f6590 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.927+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f32140422b0 con 0x7f32240f6590 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.927+0000 7f3228de2700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 0x7f32100335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:38.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.928+0000 7f3228de2700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 0x7f32100335a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f32200096a0 tx=0x7f3220006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:38.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:38.932+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f321406b6b0 con 0x7f32240f6590 2023-11-16T14:03:39.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.142+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f322406b140 con 0x7f32240f6590 2023-11-16T14:03:39.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.143+0000 7f3219ffb700 1 -- 172.21.15.112:0/3399928555 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3214026300 con 0x7f32240f6590 2023-11-16T14:03:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:39.145 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 msgr2=0x7f32100335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 0x7f32100335a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f32200096a0 tx=0x7f3220006b40).stop 2023-11-16T14:03:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 msgr2=0x7f32241089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3214004170 tx=0x7f3214009ad0).stop 2023-11-16T14:03:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 shutdown_connections 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f32100310e0 0x7f32100335a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 --2- 172.21.15.112:0/3399928555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32240f6590 0x7f32241089b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.144+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 >> 172.21.15.112:0/3399928555 conn(0x7f32240f1cd0 msgr2=0x7f32240f2970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.145+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 shutdown_connections 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:39.145+0000 7f322b847700 1 -- 172.21.15.112:0/3399928555 wait complete. 2023-11-16T14:03:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:38.855475+0000 mgr.smithi061.zcjido (mgr.14156) 64 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:39.143484+0000 mon.smithi061 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.112:0/3399928555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:40.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:40.518 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:41.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 -- 172.21.15.112:0/2657184655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 msgr2=0x7f3fc40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/2657184655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc40f58a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3fb8009fb0 tx=0x7f3fb8009b30).stop 2023-11-16T14:03:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 -- 172.21.15.112:0/2657184655 shutdown_connections 2023-11-16T14:03:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/2657184655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc40f58a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 -- 172.21.15.112:0/2657184655 >> 172.21.15.112:0/2657184655 conn(0x7f3fc40f0be0 msgr2=0x7f3fc40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 -- 172.21.15.112:0/2657184655 shutdown_connections 2023-11-16T14:03:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.582+0000 7f3fc8f18700 1 -- 172.21.15.112:0/2657184655 wait complete. 2023-11-16T14:03:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc8f18700 1 Processor -- start 2023-11-16T14:03:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc8f18700 1 -- start start 2023-11-16T14:03:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc8f18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc8f18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4107300 con 0x7f3fc40f5480 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57364/0 (socket says 172.21.15.112:57364) 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc259c700 1 -- 172.21.15.112:0/3901805974 learned_addr learned my addr 172.21.15.112:0/3901805974 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc259c700 1 -- 172.21.15.112:0/3901805974 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8005040 con 0x7f3fc40f5480 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.583+0000 7f3fc259c700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3fb80089f0 tx=0x7f3fb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb8009320 con 0x7f3fc40f5480 2023-11-16T14:03:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fb8013ec0 con 0x7f3fc40f5480 2023-11-16T14:03:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc4060300 con 0x7f3fc40f5480 2023-11-16T14:03:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb8014d30 con 0x7f3fc40f5480 2023-11-16T14:03:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc410be00 con 0x7f3fc40f5480 2023-11-16T14:03:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.584+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fb80095f0 con 0x7f3fc40f5480 2023-11-16T14:03:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.585+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc4045bb0 con 0x7f3fc40f5480 2023-11-16T14:03:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.585+0000 7f3fb2ffd700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.585+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fb80424a0 con 0x7f3fc40f5480 2023-11-16T14:03:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.585+0000 7f3fc1d9b700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.585+0000 7f3fc1d9b700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3fb40096a0 tx=0x7f3fb4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.590+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fb801d980 con 0x7f3fc40f5480 2023-11-16T14:03:41.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.798+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fc40faf70 con 0x7f3fc40f5480 2023-11-16T14:03:41.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.799+0000 7f3fb2ffd700 1 -- 172.21.15.112:0/3901805974 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fb801d380 con 0x7f3fc40f5480 2023-11-16T14:03:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:41.801 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 msgr2=0x7f3fac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3fb40096a0 tx=0x7f3fb4006b40).stop 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 msgr2=0x7f3fc410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3fb80089f0 tx=0x7f3fb800b040).stop 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 shutdown_connections 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 --2- 172.21.15.112:0/3901805974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc40f5480 0x7f3fc410b910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 >> 172.21.15.112:0/3901805974 conn(0x7f3fc40f0be0 msgr2=0x7f3fc40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 shutdown_connections 2023-11-16T14:03:41.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:41.801+0000 7f3fc8f18700 1 -- 172.21.15.112:0/3901805974 wait complete. 2023-11-16T14:03:41.804 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:40.855792+0000 mgr.smithi061.zcjido (mgr.14156) 65 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:41.799770+0000 mon.smithi061 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.112:0/3901805974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:43.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:43.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:42.856121+0000 mgr.smithi061.zcjido (mgr.14156) 66 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:44.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 -- 172.21.15.112:0/4211619207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 msgr2=0x7fbf200f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/4211619207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf200f58d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbf08009fb0 tx=0x7fbf08009b30).stop 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 -- 172.21.15.112:0/4211619207 shutdown_connections 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/4211619207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf200f58d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 -- 172.21.15.112:0/4211619207 >> 172.21.15.112:0/4211619207 conn(0x7fbf200f0c10 msgr2=0x7fbf200f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 -- 172.21.15.112:0/4211619207 shutdown_connections 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.228+0000 7fbf24c0d700 1 -- 172.21.15.112:0/4211619207 wait complete. 2023-11-16T14:03:44.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.229+0000 7fbf24c0d700 1 Processor -- start 2023-11-16T14:03:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.229+0000 7fbf24c0d700 1 -- start start 2023-11-16T14:03:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.229+0000 7fbf24c0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.229+0000 7fbf24c0d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20105b90 con 0x7fbf200f54b0 2023-11-16T14:03:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf1e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:44.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf1e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57370/0 (socket says 172.21.15.112:57370) 2023-11-16T14:03:44.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf1e59c700 1 -- 172.21.15.112:0/1671472821 learned_addr learned my addr 172.21.15.112:0/1671472821 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:44.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf1e59c700 1 -- 172.21.15.112:0/1671472821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08005040 con 0x7fbf200f54b0 2023-11-16T14:03:44.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf1e59c700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbf080089f0 tx=0x7fbf0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:44.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf08009320 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf20105d90 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbf08013ec0 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf08014d30 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.230+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf20106220 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.231+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbf080095f0 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.231+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf20045bb0 con 0x7fbf200f54b0 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.231+0000 7fbf16ffd700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 0x7fbf0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:44.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.232+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbf080423e0 con 0x7fbf200f54b0 2023-11-16T14:03:44.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.232+0000 7fbf1dd9b700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 0x7fbf0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:44.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.232+0000 7fbf1dd9b700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 0x7fbf0c033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbf100096a0 tx=0x7fbf10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:44.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.236+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbf0801d7f0 con 0x7fbf200f54b0 2023-11-16T14:03:44.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.445+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf201070d0 con 0x7fbf200f54b0 2023-11-16T14:03:44.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.446+0000 7fbf16ffd700 1 -- 172.21.15.112:0/1671472821 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf0801d7f0 con 0x7fbf200f54b0 2023-11-16T14:03:44.448 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:44.449 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:44.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 msgr2=0x7fbf0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:44.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 0x7fbf0c033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbf100096a0 tx=0x7fbf10006b40).stop 2023-11-16T14:03:44.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 msgr2=0x7fbf201056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:44.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbf080089f0 tx=0x7fbf0800b040).stop 2023-11-16T14:03:44.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 shutdown_connections 2023-11-16T14:03:44.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbf0c031090 0x7fbf0c033550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:44.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 --2- 172.21.15.112:0/1671472821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf200f54b0 0x7fbf201056a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:44.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 >> 172.21.15.112:0/1671472821 conn(0x7fbf200f0c10 msgr2=0x7fbf200f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:44.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 shutdown_connections 2023-11-16T14:03:44.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:44.447+0000 7fbf24c0d700 1 -- 172.21.15.112:0/1671472821 wait complete. 2023-11-16T14:03:44.452 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:45.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:44 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:44.446266+0000 mon.smithi061 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.112:0/1671472821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:45.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:45.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:46.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:44.856466+0000 mgr.smithi061.zcjido (mgr.14156) 67 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:46.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.937+0000 7ff8d0b90700 1 -- 172.21.15.112:0/531510765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 msgr2=0x7ff8cc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:46.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.937+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/531510765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc0f58a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff8b4009fb0 tx=0x7ff8b4009b30).stop 2023-11-16T14:03:46.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- 172.21.15.112:0/531510765 shutdown_connections 2023-11-16T14:03:46.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/531510765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc0f58a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- 172.21.15.112:0/531510765 >> 172.21.15.112:0/531510765 conn(0x7ff8cc0f0be0 msgr2=0x7ff8cc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- 172.21.15.112:0/531510765 shutdown_connections 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- 172.21.15.112:0/531510765 wait complete. 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 Processor -- start 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- start start 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8d0b90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc106cd0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.938+0000 7ff8ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51024/0 (socket says 172.21.15.112:51024) 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8ca59c700 1 -- 172.21.15.112:0/80542870 learned_addr learned my addr 172.21.15.112:0/80542870 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:46.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8ca59c700 1 -- 172.21.15.112:0/80542870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4005040 con 0x7ff8cc0f5480 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8ca59c700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff8b40089f0 tx=0x7ff8b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b4009320 con 0x7ff8cc0f5480 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc106ed0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff8b4013ec0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b4014d30 con 0x7ff8cc0f5480 2023-11-16T14:03:46.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.939+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc107360 con 0x7ff8cc0f5480 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.940+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff8b40095f0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.940+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc045bb0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.940+0000 7ff8c2ffd700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 0x7ff8b80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.940+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff8b40424a0 con 0x7ff8cc0f5480 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.940+0000 7ff8c9d9b700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 0x7ff8b80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:46.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.941+0000 7ff8c9d9b700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 0x7ff8b80335a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff8bc0096a0 tx=0x7ff8bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:46.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:46.945+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff8b401dd70 con 0x7ff8cc0f5480 2023-11-16T14:03:47.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.156+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8cc108210 con 0x7ff8cc0f5480 2023-11-16T14:03:47.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.157+0000 7ff8c2ffd700 1 -- 172.21.15.112:0/80542870 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff8b401d380 con 0x7ff8cc0f5480 2023-11-16T14:03:47.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:47.159 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:47.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 msgr2=0x7ff8b80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 0x7ff8b80335a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff8bc0096a0 tx=0x7ff8bc006b40).stop 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 msgr2=0x7ff8cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff8b40089f0 tx=0x7ff8b400b040).stop 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 shutdown_connections 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff8b80310e0 0x7ff8b80335a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 --2- 172.21.15.112:0/80542870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8cc0f5480 0x7ff8cc1067e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 >> 172.21.15.112:0/80542870 conn(0x7ff8cc0f0be0 msgr2=0x7ff8cc05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 shutdown_connections 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:47.159+0000 7ff8d0b90700 1 -- 172.21.15.112:0/80542870 wait complete. 2023-11-16T14:03:47.161 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:46.856839+0000 mgr.smithi061.zcjido (mgr.14156) 68 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:47.157342+0000 mon.smithi061 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.112:0/80542870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:48.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:48.520 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:49.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 -- 172.21.15.112:0/336670211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 msgr2=0x7fb34c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:49.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 --2- 172.21.15.112:0/336670211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c0f9160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb334009fb0 tx=0x7fb334009b30).stop 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 -- 172.21.15.112:0/336670211 shutdown_connections 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 --2- 172.21.15.112:0/336670211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c0f9160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 -- 172.21.15.112:0/336670211 >> 172.21.15.112:0/336670211 conn(0x7fb34c0f0be0 msgr2=0x7fb34c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 -- 172.21.15.112:0/336670211 shutdown_connections 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 -- 172.21.15.112:0/336670211 wait complete. 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.561+0000 7fb3520aa700 1 Processor -- start 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb3520aa700 1 -- start start 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb3520aa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:49.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb3520aa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c106c40 con 0x7fb34c060360 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb34b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb34b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51044/0 (socket says 172.21.15.112:51044) 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb34b7fe700 1 -- 172.21.15.112:0/3222341090 learned_addr learned my addr 172.21.15.112:0/3222341090 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb34b7fe700 1 -- 172.21.15.112:0/3222341090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334005040 con 0x7fb34c060360 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.562+0000 7fb34b7fe700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb334009f80 tx=0x7fb33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb334009320 con 0x7fb34c060360 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c106e40 con 0x7fb34c060360 2023-11-16T14:03:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb334013680 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb33401eda0 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c1072d0 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.563+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb33401e420 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.564+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c045bb0 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.564+0000 7fb343fff700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 0x7fb338033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.564+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb334042440 con 0x7fb34c060360 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.564+0000 7fb34affd700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 0x7fb338033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.564+0000 7fb34affd700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 0x7fb338033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.569+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb33401d380 con 0x7fb34c060360 2023-11-16T14:03:49.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.777+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb34c108180 con 0x7fb34c060360 2023-11-16T14:03:49.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.778+0000 7fb343fff700 1 -- 172.21.15.112:0/3222341090 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb334004490 con 0x7fb34c060360 2023-11-16T14:03:49.779 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:49.780 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:49.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 msgr2=0x7fb338033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:49.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 0x7fb338033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).stop 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 msgr2=0x7fb34c106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb334009f80 tx=0x7fb33400b040).stop 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 shutdown_connections 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb338031090 0x7fb338033550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 --2- 172.21.15.112:0/3222341090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb34c060360 0x7fb34c106750 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.779+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 >> 172.21.15.112:0/3222341090 conn(0x7fb34c0f0be0 msgr2=0x7fb34c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.780+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 shutdown_connections 2023-11-16T14:03:49.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:49.780+0000 7fb3520aa700 1 -- 172.21.15.112:0/3222341090 wait complete. 2023-11-16T14:03:49.783 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:48.857186+0000 mgr.smithi061.zcjido (mgr.14156) 69 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:49.778215+0000 mon.smithi061 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.112:0/3222341090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:51.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:51.116 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 -- 172.21.15.112:0/2667104340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 msgr2=0x7f58c00f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 --2- 172.21.15.112:0/2667104340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c00f4c60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f58b0009fb0 tx=0x7f58b0009b30).stop 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 -- 172.21.15.112:0/2667104340 shutdown_connections 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 --2- 172.21.15.112:0/2667104340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c00f4c60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 -- 172.21.15.112:0/2667104340 >> 172.21.15.112:0/2667104340 conn(0x7f58c00f0360 msgr2=0x7f58c00f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 -- 172.21.15.112:0/2667104340 shutdown_connections 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.216+0000 7f58c4944700 1 -- 172.21.15.112:0/2667104340 wait complete. 2023-11-16T14:03:52.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58c4944700 1 Processor -- start 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58c4944700 1 -- start start 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58c4944700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58c4944700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c0106cb0 con 0x7f58c00f4840 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58be59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58be59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51070/0 (socket says 172.21.15.112:51070) 2023-11-16T14:03:52.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.217+0000 7f58be59c700 1 -- 172.21.15.112:0/3955263105 learned_addr learned my addr 172.21.15.112:0/3955263105 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:52.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58be59c700 1 -- 172.21.15.112:0/3955263105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0005040 con 0x7f58c00f4840 2023-11-16T14:03:52.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58be59c700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f58b00089f0 tx=0x7f58b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:52.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b0009320 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c0106eb0 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f58b0013ec0 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b0014d30 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.218+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c0107340 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.219+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f58b00095f0 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.219+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c0045bb0 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.219+0000 7f58aeffd700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 0x7f58a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.219+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f58b0042420 con 0x7f58c00f4840 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.219+0000 7f58bdd9b700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 0x7f58a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:52.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.220+0000 7f58bdd9b700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 0x7f58a8033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f58b40096a0 tx=0x7f58b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:52.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.224+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f58b001d7f0 con 0x7f58c00f4840 2023-11-16T14:03:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:50.857538+0000 mgr.smithi061.zcjido (mgr.14156) 70 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:52.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.431+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58c0108190 con 0x7f58c00f4840 2023-11-16T14:03:52.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.432+0000 7f58aeffd700 1 -- 172.21.15.112:0/3955263105 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58b000fe40 con 0x7f58c00f4840 2023-11-16T14:03:52.434 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:52.434 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:52.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 msgr2=0x7f58a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 0x7f58a8033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f58b40096a0 tx=0x7f58b4006b40).stop 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 msgr2=0x7f58c01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f58b00089f0 tx=0x7f58b000b040).stop 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 shutdown_connections 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f58a8031090 0x7f58a8033550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 --2- 172.21.15.112:0/3955263105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58c00f4840 0x7f58c01067c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 >> 172.21.15.112:0/3955263105 conn(0x7f58c00f0360 msgr2=0x7f58c00f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 shutdown_connections 2023-11-16T14:03:52.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:52.434+0000 7f58c4944700 1 -- 172.21.15.112:0/3955263105 wait complete. 2023-11-16T14:03:52.437 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:53.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:52.432553+0000 mon.smithi061 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.112:0/3955263105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:53.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:53.821 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:54.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:52.857873+0000 mgr.smithi061.zcjido (mgr.14156) 71 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:54.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 -- 172.21.15.112:0/804115520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 msgr2=0x7f47780f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:54.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 --2- 172.21.15.112:0/804115520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f47780f58d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4768009fb0 tx=0x7f4768009b30).stop 2023-11-16T14:03:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 -- 172.21.15.112:0/804115520 shutdown_connections 2023-11-16T14:03:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 --2- 172.21.15.112:0/804115520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f47780f58d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 -- 172.21.15.112:0/804115520 >> 172.21.15.112:0/804115520 conn(0x7f47780f0c10 msgr2=0x7f47780f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 -- 172.21.15.112:0/804115520 shutdown_connections 2023-11-16T14:03:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.912+0000 7f477cce6700 1 -- 172.21.15.112:0/804115520 wait complete. 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.913+0000 7f477cce6700 1 Processor -- start 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.913+0000 7f477cce6700 1 -- start start 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.913+0000 7f477cce6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.913+0000 7f477cce6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4778105bf0 con 0x7f47780f54b0 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.913+0000 7f477659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:51088/0 (socket says 172.21.15.112:51088) 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477659c700 1 -- 172.21.15.112:0/2330267654 learned_addr learned my addr 172.21.15.112:0/2330267654 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477659c700 1 -- 172.21.15.112:0/2330267654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4768005040 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477659c700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f47680089f0 tx=0x7f476800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768009320 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4778105df0 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4768013ec0 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768014d30 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.914+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4778106280 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.915+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f47680095f0 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.915+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4778045bb0 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.915+0000 7f4766ffd700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 0x7f4760033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.915+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47680424a0 con 0x7f47780f54b0 2023-11-16T14:03:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.915+0000 7f4775d9b700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 0x7f4760033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.916+0000 7f4775d9b700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 0x7f4760033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f476c0096a0 tx=0x7f476c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:54.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:54.920+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f476801d7f0 con 0x7f47780f54b0 2023-11-16T14:03:55.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.126+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4778107130 con 0x7f47780f54b0 2023-11-16T14:03:55.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.127+0000 7f4766ffd700 1 -- 172.21.15.112:0/2330267654 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f476801d7f0 con 0x7f47780f54b0 2023-11-16T14:03:55.130 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:55.130 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.128+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 msgr2=0x7f4760033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.128+0000 7f477cce6700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 0x7f4760033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f476c0096a0 tx=0x7f476c006b40).stop 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.128+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 msgr2=0x7f4778105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.128+0000 7f477cce6700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f47680089f0 tx=0x7f476800b040).stop 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 shutdown_connections 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4760031090 0x7f4760033550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 --2- 172.21.15.112:0/2330267654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47780f54b0 0x7f4778105700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 >> 172.21.15.112:0/2330267654 conn(0x7f47780f0c10 msgr2=0x7f47780f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 shutdown_connections 2023-11-16T14:03:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:55.129+0000 7f477cce6700 1 -- 172.21.15.112:0/2330267654 wait complete. 2023-11-16T14:03:55.133 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:54.858252+0000 mgr.smithi061.zcjido (mgr.14156) 72 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:55.128522+0000 mon.smithi061 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.112:0/2330267654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:56.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:56.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/867081148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 msgr2=0x7fcb980f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/867081148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f4d10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fcb8c009fb0 tx=0x7fcb8c009b30).stop 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/867081148 shutdown_connections 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/867081148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f4d10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/867081148 >> 172.21.15.112:0/867081148 conn(0x7fcb980f0390 msgr2=0x7fcb980f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/867081148 shutdown_connections 2023-11-16T14:03:57.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.579+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/867081148 wait complete. 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ecfe700 1 Processor -- start 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ecfe700 1 -- start start 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ecfe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ecfe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980f5960 con 0x7fcb980f48f0 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ca9a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ca9a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:47984/0 (socket says 172.21.15.112:47984) 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.580+0000 7fcb9ca9a700 1 -- 172.21.15.112:0/3013631215 learned_addr learned my addr 172.21.15.112:0/3013631215 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb9ca9a700 1 -- 172.21.15.112:0/3013631215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c005040 con 0x7fcb980f48f0 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb9ca9a700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fcb8c009f80 tx=0x7fcb8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c009320 con 0x7fcb980f48f0 2023-11-16T14:03:57.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb980f5b00 con 0x7fcb980f48f0 2023-11-16T14:03:57.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcb8c013680 con 0x7fcb980f48f0 2023-11-16T14:03:57.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c01eda0 con 0x7fcb980f48f0 2023-11-16T14:03:57.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.581+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb980f5f90 con 0x7fcb980f48f0 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.582+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcb8c01e420 con 0x7fcb980f48f0 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.582+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb980591e0 con 0x7fcb980f48f0 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.582+0000 7fcb957fa700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 0x7fcb88033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.582+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcb8c042380 con 0x7fcb980f48f0 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.582+0000 7fcb97fff700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 0x7fcb88033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:03:57.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.583+0000 7fcb97fff700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 0x7fcb88033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb840096a0 tx=0x7fcb84006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:03:57.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.587+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb8c014cc0 con 0x7fcb980f48f0 2023-11-16T14:03:57.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.798+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb98045bb0 con 0x7fcb980f48f0 2023-11-16T14:03:57.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.798+0000 7fcb957fa700 1 -- 172.21.15.112:0/3013631215 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb8c004490 con 0x7fcb980f48f0 2023-11-16T14:03:57.801 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:03:57.802 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:03:57.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 msgr2=0x7fcb88033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 0x7fcb88033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb840096a0 tx=0x7fcb84006b40).stop 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 msgr2=0x7fcb980f74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fcb8c009f80 tx=0x7fcb8c00b040).stop 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 shutdown_connections 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcb88031090 0x7fcb88033550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 --2- 172.21.15.112:0/3013631215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb980f48f0 0x7fcb980f74e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:03:57.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 >> 172.21.15.112:0/3013631215 conn(0x7fcb980f0390 msgr2=0x7fcb980f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:03:57.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 shutdown_connections 2023-11-16T14:03:57.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:03:57.800+0000 7fcb9ecfe700 1 -- 172.21.15.112:0/3013631215 wait complete. 2023-11-16T14:03:57.805 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:03:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:56.858623+0000 mgr.smithi061.zcjido (mgr.14156) 73 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:03:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:03:57.799855+0000 mon.smithi061 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.112:0/3013631215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:03:59.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:03:59.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:03:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:03:58.859018+0000 mgr.smithi061.zcjido (mgr.14156) 74 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:00.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.287+0000 7fe8c72ce700 1 -- 172.21.15.112:0/1894315725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 msgr2=0x7fe8c00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:00.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.287+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/1894315725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c00f6800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe8b0009fb0 tx=0x7fe8b0009b30).stop 2023-11-16T14:04:00.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- 172.21.15.112:0/1894315725 shutdown_connections 2023-11-16T14:04:00.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/1894315725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c00f6800 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- 172.21.15.112:0/1894315725 >> 172.21.15.112:0/1894315725 conn(0x7fe8c00f1b40 msgr2=0x7fe8c00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- 172.21.15.112:0/1894315725 shutdown_connections 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- 172.21.15.112:0/1894315725 wait complete. 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 Processor -- start 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- start start 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c72ce700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c0069f00 con 0x7fe8c00f63e0 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.288+0000 7fe8c62cc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c62cc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48008/0 (socket says 172.21.15.112:48008) 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c62cc700 1 -- 172.21.15.112:0/320551558 learned_addr learned my addr 172.21.15.112:0/320551558 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c62cc700 1 -- 172.21.15.112:0/320551558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0005040 con 0x7fe8c00f63e0 2023-11-16T14:04:00.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c62cc700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe8b00089f0 tx=0x7fe8b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0009320 con 0x7fe8c00f63e0 2023-11-16T14:04:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe8b0013ec0 con 0x7fe8c00f63e0 2023-11-16T14:04:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c006a0a0 con 0x7fe8c00f63e0 2023-11-16T14:04:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0014d30 con 0x7fe8c00f63e0 2023-11-16T14:04:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.289+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c006a530 con 0x7fe8c00f63e0 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.290+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe8b00095f0 con 0x7fe8c00f63e0 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.290+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c00591e0 con 0x7fe8c00f63e0 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.290+0000 7fe8b6ffd700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 0x7fe8ac0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.290+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8b0042450 con 0x7fe8c00f63e0 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.290+0000 7fe8c5acb700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 0x7fe8ac0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.291+0000 7fe8c5acb700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 0x7fe8ac0335a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe8bc0096a0 tx=0x7fe8bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:00.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.295+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe8b001d980 con 0x7fe8c00f63e0 2023-11-16T14:04:00.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.503+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8c00fbed0 con 0x7fe8c00f63e0 2023-11-16T14:04:00.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.503+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/320551558 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8b001d380 con 0x7fe8c00f63e0 2023-11-16T14:04:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:00.506 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:00.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 msgr2=0x7fe8ac0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 0x7fe8ac0335a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe8bc0096a0 tx=0x7fe8bc006b40).stop 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 msgr2=0x7fe8c006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe8b00089f0 tx=0x7fe8b000b040).stop 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 shutdown_connections 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8ac0310e0 0x7fe8ac0335a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 --2- 172.21.15.112:0/320551558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c00f63e0 0x7fe8c006ba80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 >> 172.21.15.112:0/320551558 conn(0x7fe8c00f1b40 msgr2=0x7fe8c00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 shutdown_connections 2023-11-16T14:04:00.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:00.505+0000 7fe8c72ce700 1 -- 172.21.15.112:0/320551558 wait complete. 2023-11-16T14:04:00.509 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:01.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:00 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:00.504799+0000 mon.smithi061 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.112:0/320551558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:01.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:01.839 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:00.859330+0000 mgr.smithi061.zcjido (mgr.14156) 75 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- 172.21.15.112:0/3424070717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 msgr2=0x7f44040f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 --2- 172.21.15.112:0/3424070717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040f56f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f43f8009fb0 tx=0x7f43f8009b30).stop 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- 172.21.15.112:0/3424070717 shutdown_connections 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 --2- 172.21.15.112:0/3424070717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040f56f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- 172.21.15.112:0/3424070717 >> 172.21.15.112:0/3424070717 conn(0x7f44040f0a10 msgr2=0x7f44040f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- 172.21.15.112:0/3424070717 shutdown_connections 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- 172.21.15.112:0/3424070717 wait complete. 2023-11-16T14:04:02.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 Processor -- start 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.935+0000 7f4403fff700 1 -- start start 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4403fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4403fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44040fb730 con 0x7f44040f52d0 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48024/0 (socket says 172.21.15.112:48024) 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4402ffd700 1 -- 172.21.15.112:0/4017136230 learned_addr learned my addr 172.21.15.112:0/4017136230 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4402ffd700 1 -- 172.21.15.112:0/4017136230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8005040 con 0x7f44040f52d0 2023-11-16T14:04:02.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.936+0000 7f4402ffd700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f43f80089f0 tx=0x7f43f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:02.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f8009320 con 0x7f44040f52d0 2023-11-16T14:04:02.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f4403fff700 1 -- 172.21.15.112:0/4017136230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44040fb8d0 con 0x7f44040f52d0 2023-11-16T14:04:02.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f43f8013ec0 con 0x7f44040f52d0 2023-11-16T14:04:02.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f4403fff700 1 -- 172.21.15.112:0/4017136230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44040fbd60 con 0x7f44040f52d0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f8014d30 con 0x7f44040f52d0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.937+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f43f80263a0 con 0x7f44040f52d0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.938+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc004fa0 con 0x7f44040f52d0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.938+0000 7f43eb7fe700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 0x7f43ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.938+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f43f8042630 con 0x7f44040f52d0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.938+0000 7f44027fc700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 0x7f43ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:02.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.938+0000 7f44027fc700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 0x7f43ec033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f43f40096a0 tx=0x7f43f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:02.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:02.943+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f43f8012330 con 0x7f44040f52d0 2023-11-16T14:04:03.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.153+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43dc005670 con 0x7f44040f52d0 2023-11-16T14:04:03.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.154+0000 7f43eb7fe700 1 -- 172.21.15.112:0/4017136230 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43f801db70 con 0x7f44040f52d0 2023-11-16T14:04:03.157 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:03.157 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:03.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 msgr2=0x7f43ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:03.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 0x7f43ec033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f43f40096a0 tx=0x7f43f4006b40).stop 2023-11-16T14:04:03.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 msgr2=0x7f44040fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f43f80089f0 tx=0x7f43f800b040).stop 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 shutdown_connections 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43ec031090 0x7f43ec033550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 --2- 172.21.15.112:0/4017136230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44040f52d0 0x7f44040fd2b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 >> 172.21.15.112:0/4017136230 conn(0x7f44040f0a10 msgr2=0x7f440405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 shutdown_connections 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:03.155+0000 7f43e97fa700 1 -- 172.21.15.112:0/4017136230 wait complete. 2023-11-16T14:04:03.159 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:02.859582+0000 mgr.smithi061.zcjido (mgr.14156) 76 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:03.155365+0000 mon.smithi061 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.112:0/4017136230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:04.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:04.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 -- 172.21.15.112:0/806117728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 msgr2=0x7f53bc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 --2- 172.21.15.112:0/806117728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc0f69b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f53ac004760 tx=0x7f53ac009ad0).stop 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 -- 172.21.15.112:0/806117728 shutdown_connections 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 --2- 172.21.15.112:0/806117728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc0f69b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 -- 172.21.15.112:0/806117728 >> 172.21.15.112:0/806117728 conn(0x7f53bc0f1cd0 msgr2=0x7f53bc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:05.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 -- 172.21.15.112:0/806117728 shutdown_connections 2023-11-16T14:04:05.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.625+0000 7f53c28f9700 1 -- 172.21.15.112:0/806117728 wait complete. 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53c28f9700 1 Processor -- start 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53c28f9700 1 -- start start 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53c28f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53c28f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc106cb0 con 0x7f53bc0f6590 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:56928/0 (socket says 172.21.15.112:56928) 2023-11-16T14:04:05.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.626+0000 7f53bbfff700 1 -- 172.21.15.112:0/2225898878 learned_addr learned my addr 172.21.15.112:0/2225898878 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53bbfff700 1 -- 172.21.15.112:0/2225898878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac009d70 con 0x7f53bc0f6590 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53bbfff700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f53ac004170 tx=0x7f53ac009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ac00e040 con 0x7f53bc0f6590 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f53ac008ef0 con 0x7f53bc0f6590 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ac014c90 con 0x7f53bc0f6590 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc106eb0 con 0x7f53bc0f6590 2023-11-16T14:04:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.627+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc107340 con 0x7f53bc0f6590 2023-11-16T14:04:05.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.628+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f53ac014420 con 0x7f53bc0f6590 2023-11-16T14:04:05.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.628+0000 7f53b8ff9700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 0x7f53a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:05.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.628+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc045bb0 con 0x7f53bc0f6590 2023-11-16T14:04:05.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.628+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f53ac0422c0 con 0x7f53bc0f6590 2023-11-16T14:04:05.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.628+0000 7f53bb7fe700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 0x7f53a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:05.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.629+0000 7f53bb7fe700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 0x7f53a8033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f53a40096a0 tx=0x7f53a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:05.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.633+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f53ac026300 con 0x7f53bc0f6590 2023-11-16T14:04:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.843+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53bc0fc170 con 0x7f53bc0f6590 2023-11-16T14:04:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.843+0000 7f53b8ff9700 1 -- 172.21.15.112:0/2225898878 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f53ac012b20 con 0x7f53bc0f6590 2023-11-16T14:04:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:05.846 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:05.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 msgr2=0x7f53a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:05.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 0x7f53a8033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f53a40096a0 tx=0x7f53a4006b40).stop 2023-11-16T14:04:05.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 msgr2=0x7f53bc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:05.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f53ac004170 tx=0x7f53ac009ad0).stop 2023-11-16T14:04:05.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 shutdown_connections 2023-11-16T14:04:05.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f53a8031090 0x7f53a8033550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:05.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 --2- 172.21.15.112:0/2225898878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53bc0f6590 0x7f53bc1067c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:05.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 >> 172.21.15.112:0/2225898878 conn(0x7f53bc0f1cd0 msgr2=0x7f53bc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:05.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 shutdown_connections 2023-11-16T14:04:05.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:05.845+0000 7f53c28f9700 1 -- 172.21.15.112:0/2225898878 wait complete. 2023-11-16T14:04:05.851 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:04.859942+0000 mgr.smithi061.zcjido (mgr.14156) 77 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:05.844877+0000 mon.smithi061 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.112:0/2225898878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:07.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:07.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:08.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.245+0000 7f8f56239700 1 -- 172.21.15.112:0/2275321658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 msgr2=0x7f8f500f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.245+0000 7f8f56239700 1 --2- 172.21.15.112:0/2275321658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f500f69b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8f44004760 tx=0x7f8f44009ad0).stop 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.245+0000 7f8f56239700 1 -- 172.21.15.112:0/2275321658 shutdown_connections 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.245+0000 7f8f56239700 1 --2- 172.21.15.112:0/2275321658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f500f69b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.245+0000 7f8f56239700 1 -- 172.21.15.112:0/2275321658 >> 172.21.15.112:0/2275321658 conn(0x7f8f500f1cd0 msgr2=0x7f8f500f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.246+0000 7f8f56239700 1 -- 172.21.15.112:0/2275321658 shutdown_connections 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.246+0000 7f8f56239700 1 -- 172.21.15.112:0/2275321658 wait complete. 2023-11-16T14:04:08.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.246+0000 7f8f56239700 1 Processor -- start 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.246+0000 7f8f56239700 1 -- start start 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f56239700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f56239700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50106cb0 con 0x7f8f500f6590 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f4f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f4f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:56952/0 (socket says 172.21.15.112:56952) 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f4f7fe700 1 -- 172.21.15.112:0/2006552852 learned_addr learned my addr 172.21.15.112:0/2006552852 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:08.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f4f7fe700 1 -- 172.21.15.112:0/2006552852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44009d70 con 0x7f8f500f6590 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f4f7fe700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8f44004170 tx=0x7f8f44009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f4400e040 con 0x7f8f500f6590 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f50106eb0 con 0x7f8f500f6590 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8f44008ef0 con 0x7f8f500f6590 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f44014c90 con 0x7f8f500f6590 2023-11-16T14:04:08.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.247+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f50107340 con 0x7f8f500f6590 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.248+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8f44014420 con 0x7f8f500f6590 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.248+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50045bb0 con 0x7f8f500f6590 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.248+0000 7f8f37fff700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 0x7f8f38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.248+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8f44042220 con 0x7f8f500f6590 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.249+0000 7f8f4effd700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 0x7f8f38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:08.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.249+0000 7f8f4effd700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 0x7f8f38033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8f400096a0 tx=0x7f8f40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:08.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.253+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8f44012d00 con 0x7f8f500f6590 2023-11-16T14:04:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:06.860282+0000 mgr.smithi061.zcjido (mgr.14156) 78 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:08.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.462+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f50108190 con 0x7f8f500f6590 2023-11-16T14:04:08.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.463+0000 7f8f37fff700 1 -- 172.21.15.112:0/2006552852 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f44012b20 con 0x7f8f500f6590 2023-11-16T14:04:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:08.467 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:08.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 msgr2=0x7f8f38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:08.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 0x7f8f38033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8f400096a0 tx=0x7f8f40006b40).stop 2023-11-16T14:04:08.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 msgr2=0x7f8f501067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:08.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8f44004170 tx=0x7f8f44009ad0).stop 2023-11-16T14:04:08.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 shutdown_connections 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8f38031090 0x7f8f38033550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 --2- 172.21.15.112:0/2006552852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f500f6590 0x7f8f501067c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 >> 172.21.15.112:0/2006552852 conn(0x7f8f500f1cd0 msgr2=0x7f8f500f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 shutdown_connections 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:08.464+0000 7f8f56239700 1 -- 172.21.15.112:0/2006552852 wait complete. 2023-11-16T14:04:08.469 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:09.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:08.464163+0000 mon.smithi061 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.112:0/2006552852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:09.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:09.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:08.860657+0000 mgr.smithi061.zcjido (mgr.14156) 79 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 -- 172.21.15.112:0/185858684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 msgr2=0x7faf980f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 --2- 172.21.15.112:0/185858684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf980f6800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7faf80009fb0 tx=0x7faf80009b30).stop 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 -- 172.21.15.112:0/185858684 shutdown_connections 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 --2- 172.21.15.112:0/185858684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf980f6800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 -- 172.21.15.112:0/185858684 >> 172.21.15.112:0/185858684 conn(0x7faf980f1b40 msgr2=0x7faf980f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 -- 172.21.15.112:0/185858684 shutdown_connections 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.925+0000 7faf97fff700 1 -- 172.21.15.112:0/185858684 wait complete. 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf97fff700 1 Processor -- start 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf97fff700 1 -- start start 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf97fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf97fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf98106b50 con 0x7faf980f63e0 2023-11-16T14:04:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf96ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:10.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf96ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:56970/0 (socket says 172.21.15.112:56970) 2023-11-16T14:04:10.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf96ffd700 1 -- 172.21.15.112:0/437806166 learned_addr learned my addr 172.21.15.112:0/437806166 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf96ffd700 1 -- 172.21.15.112:0/437806166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80005040 con 0x7faf980f63e0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.926+0000 7faf96ffd700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faf800089f0 tx=0x7faf8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.927+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf80009320 con 0x7faf980f63e0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.927+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf98106d50 con 0x7faf980f63e0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.927+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faf80013ec0 con 0x7faf980f63e0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.927+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf80014d30 con 0x7faf980f63e0 2023-11-16T14:04:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.927+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf981071e0 con 0x7faf980f63e0 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.928+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7faf800095f0 con 0x7faf980f63e0 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.928+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf98045bb0 con 0x7faf980f63e0 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.928+0000 7faf8f7fe700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 0x7faf84033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.928+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faf800423e0 con 0x7faf980f63e0 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.928+0000 7faf967fc700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 0x7faf84033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:10.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.929+0000 7faf967fc700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 0x7faf84033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7faf880096a0 tx=0x7faf88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:10.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:10.933+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faf8001d980 con 0x7faf980f63e0 2023-11-16T14:04:11.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.141+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf98108090 con 0x7faf980f63e0 2023-11-16T14:04:11.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.141+0000 7faf8f7fe700 1 -- 172.21.15.112:0/437806166 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf8001d380 con 0x7faf980f63e0 2023-11-16T14:04:11.144 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:11.144 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 msgr2=0x7faf84033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 0x7faf84033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7faf880096a0 tx=0x7faf88006b40).stop 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 msgr2=0x7faf98106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faf800089f0 tx=0x7faf8000b040).stop 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 shutdown_connections 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faf84031090 0x7faf84033550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 --2- 172.21.15.112:0/437806166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980f63e0 0x7faf98106660 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 >> 172.21.15.112:0/437806166 conn(0x7faf980f1b40 msgr2=0x7faf980f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:11.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 shutdown_connections 2023-11-16T14:04:11.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:11.143+0000 7faf97fff700 1 -- 172.21.15.112:0/437806166 wait complete. 2023-11-16T14:04:11.147 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:10.861034+0000 mgr.smithi061.zcjido (mgr.14156) 80 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:11.142759+0000 mon.smithi061 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.112:0/437806166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:12.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:12.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:13.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.594+0000 7fade8f8a700 1 -- 172.21.15.112:0/600173420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 msgr2=0x7fade40f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:13.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.594+0000 7fade8f8a700 1 --2- 172.21.15.112:0/600173420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade40f6800 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fadd4009fb0 tx=0x7fadd4009b30).stop 2023-11-16T14:04:13.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.594+0000 7fade8f8a700 1 -- 172.21.15.112:0/600173420 shutdown_connections 2023-11-16T14:04:13.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.594+0000 7fade8f8a700 1 --2- 172.21.15.112:0/600173420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade40f6800 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:13.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 -- 172.21.15.112:0/600173420 >> 172.21.15.112:0/600173420 conn(0x7fade40f1b40 msgr2=0x7fade40f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:13.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 -- 172.21.15.112:0/600173420 shutdown_connections 2023-11-16T14:04:13.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 -- 172.21.15.112:0/600173420 wait complete. 2023-11-16T14:04:13.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 Processor -- start 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 -- start start 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.595+0000 7fade8f8a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade4106b10 con 0x7fade40f63e0 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:56990/0 (socket says 172.21.15.112:56990) 2023-11-16T14:04:13.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade37fe700 1 -- 172.21.15.112:0/1881365496 learned_addr learned my addr 172.21.15.112:0/1881365496 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade37fe700 1 -- 172.21.15.112:0/1881365496 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4005040 con 0x7fade40f63e0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade37fe700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fadd40089f0 tx=0x7fadd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadd4009320 con 0x7fade40f63e0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade4106d10 con 0x7fade40f63e0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fadd4013ec0 con 0x7fade40f63e0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade41071a0 con 0x7fade40f63e0 2023-11-16T14:04:13.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.596+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadd4014d30 con 0x7fade40f63e0 2023-11-16T14:04:13.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.597+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fadd40095f0 con 0x7fade40f63e0 2023-11-16T14:04:13.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.597+0000 7fadcbfff700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 0x7fadcc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:13.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.597+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fadd4042500 con 0x7fade40f63e0 2023-11-16T14:04:13.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.597+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade4045bb0 con 0x7fade40f63e0 2023-11-16T14:04:13.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.597+0000 7fade2ffd700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 0x7fadcc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:13.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.598+0000 7fade2ffd700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 0x7fadcc033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fadd80096a0 tx=0x7fadd8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:13.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.602+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fadd401dbe0 con 0x7fade40f63e0 2023-11-16T14:04:13.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.811+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fade40fbfc0 con 0x7fade40f63e0 2023-11-16T14:04:13.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.812+0000 7fadcbfff700 1 -- 172.21.15.112:0/1881365496 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fadd402b020 con 0x7fade40f63e0 2023-11-16T14:04:13.815 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:13.815 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 msgr2=0x7fadcc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 0x7fadcc033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fadd80096a0 tx=0x7fadd8006b40).stop 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 msgr2=0x7fade4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fadd40089f0 tx=0x7fadd400b040).stop 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 shutdown_connections 2023-11-16T14:04:13.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fadcc031090 0x7fadcc033550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:13.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 --2- 172.21.15.112:0/1881365496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade40f63e0 0x7fade4106620 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:13.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 >> 172.21.15.112:0/1881365496 conn(0x7fade40f1b40 msgr2=0x7fade40f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:13.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 shutdown_connections 2023-11-16T14:04:13.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:13.813+0000 7fade8f8a700 1 -- 172.21.15.112:0/1881365496 wait complete. 2023-11-16T14:04:13.817 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:12.861305+0000 mgr.smithi061.zcjido (mgr.14156) 81 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:15.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:13.813190+0000 mon.smithi061 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.112:0/1881365496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:15.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:15.175 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:16.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:14.861675+0000 mgr.smithi061.zcjido (mgr.14156) 82 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:16.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 -- 172.21.15.112:0/2375936867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200f5480 msgr2=0x7f02200f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:16.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 --2- 172.21.15.112:0/2375936867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200f5480 0x7f02200f58a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0214009fb0 tx=0x7f0214009b30).stop 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 -- 172.21.15.112:0/2375936867 shutdown_connections 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 --2- 172.21.15.112:0/2375936867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200f5480 0x7f02200f58a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 -- 172.21.15.112:0/2375936867 >> 172.21.15.112:0/2375936867 conn(0x7f02200f0be0 msgr2=0x7f02200f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 -- 172.21.15.112:0/2375936867 shutdown_connections 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.216+0000 7f02271d5700 1 -- 172.21.15.112:0/2375936867 wait complete. 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f02271d5700 1 Processor -- start 2023-11-16T14:04:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f02271d5700 1 -- start start 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f02271d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f02271d5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220109af0 con 0x7f02200602e0 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f0224f71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f0224f71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38268/0 (socket says 172.21.15.112:38268) 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f0224f71700 1 -- 172.21.15.112:0/3468949651 learned_addr learned my addr 172.21.15.112:0/3468949651 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.217+0000 7f0224f71700 1 -- 172.21.15.112:0/3468949651 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214005040 con 0x7f02200602e0 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f0224f71700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0214009880 tx=0x7f0214009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0214013680 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0220109c30 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f02140095f0 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0214014e40 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.218+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022010a060 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0214014420 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0220045bb0 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f021d7fa700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 0x7f0210033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0214042480 con 0x7f02200602e0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f021ffff700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 0x7f0210033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:16.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.219+0000 7f021ffff700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 0x7f0210033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f020c0096a0 tx=0x7f020c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.224+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0214025ae0 con 0x7f02200602e0 2023-11-16T14:04:16.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.430+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02200faf70 con 0x7f02200602e0 2023-11-16T14:04:16.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.430+0000 7f021d7fa700 1 -- 172.21.15.112:0/3468949651 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0214025ae0 con 0x7f02200602e0 2023-11-16T14:04:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:16.433 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 msgr2=0x7f0210033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 0x7f0210033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f020c0096a0 tx=0x7f020c006b40).stop 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 msgr2=0x7f0220109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0214009880 tx=0x7f0214009b30).stop 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 shutdown_connections 2023-11-16T14:04:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0210031090 0x7f0210033550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 --2- 172.21.15.112:0/3468949651 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02200602e0 0x7f0220109600 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 >> 172.21.15.112:0/3468949651 conn(0x7f02200f0be0 msgr2=0x7f02200f1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 shutdown_connections 2023-11-16T14:04:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:16.432+0000 7f02271d5700 1 -- 172.21.15.112:0/3468949651 wait complete. 2023-11-16T14:04:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:17.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:16 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:16.430879+0000 mon.smithi061 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.112:0/3468949651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:17.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:17.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:18.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:16.861996+0000 mgr.smithi061.zcjido (mgr.14156) 83 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.857+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/3290669614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 msgr2=0x7ff0bc0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.857+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/3290669614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc0f6800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff0ac009fb0 tx=0x7ff0ac009b30).stop 2023-11-16T14:04:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/3290669614 shutdown_connections 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/3290669614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc0f6800 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/3290669614 >> 172.21.15.112:0/3290669614 conn(0x7ff0bc0f1b40 msgr2=0x7ff0bc0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/3290669614 shutdown_connections 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/3290669614 wait complete. 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 Processor -- start 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- start start 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.858+0000 7ff0c0ae1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc106b10 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38278/0 (socket says 172.21.15.112:38278) 2023-11-16T14:04:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0bb7fe700 1 -- 172.21.15.112:0/1314726166 learned_addr learned my addr 172.21.15.112:0/1314726166 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0bb7fe700 1 -- 172.21.15.112:0/1314726166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac005040 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0bb7fe700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff0ac0089f0 tx=0x7ff0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac009320 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc106d10 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff0ac013ec0 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac014d30 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.859+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc1071a0 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.860+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff0ac0095f0 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.860+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc045bb0 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.860+0000 7ff0a3fff700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 0x7ff0a40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.860+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff0ac0424a0 con 0x7ff0bc0f63e0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.861+0000 7ff0baffd700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 0x7ff0a40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.861+0000 7ff0baffd700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 0x7ff0a40335a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ff0b00096a0 tx=0x7ff0b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:18.865+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff0ac0263a0 con 0x7ff0bc0f63e0 2023-11-16T14:04:19.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.075+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0bc108050 con 0x7ff0bc0f63e0 2023-11-16T14:04:19.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.076+0000 7ff0a3fff700 1 -- 172.21.15.112:0/1314726166 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0ac01d380 con 0x7ff0bc0f63e0 2023-11-16T14:04:19.079 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:19.079 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:19.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 msgr2=0x7ff0a40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:19.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 0x7ff0a40335a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ff0b00096a0 tx=0x7ff0b0006b40).stop 2023-11-16T14:04:19.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 msgr2=0x7ff0bc106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:19.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff0ac0089f0 tx=0x7ff0ac00b040).stop 2023-11-16T14:04:19.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 shutdown_connections 2023-11-16T14:04:19.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff0a40310e0 0x7ff0a40335a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:19.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 --2- 172.21.15.112:0/1314726166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0bc0f63e0 0x7ff0bc106620 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:19.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 >> 172.21.15.112:0/1314726166 conn(0x7ff0bc0f1b40 msgr2=0x7ff0bc0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:19.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 shutdown_connections 2023-11-16T14:04:19.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:19.078+0000 7ff0c0ae1700 1 -- 172.21.15.112:0/1314726166 wait complete. 2023-11-16T14:04:19.083 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:18.862247+0000 mgr.smithi061.zcjido (mgr.14156) 84 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:19.076385+0000 mon.smithi061 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.112:0/1314726166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:20.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:20.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- 172.21.15.112:0/2232304224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 msgr2=0x7f1ad00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 --2- 172.21.15.112:0/2232304224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad00f6800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1ac4009fb0 tx=0x7f1ac4009b30).stop 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- 172.21.15.112:0/2232304224 shutdown_connections 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 --2- 172.21.15.112:0/2232304224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad00f6800 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- 172.21.15.112:0/2232304224 >> 172.21.15.112:0/2232304224 conn(0x7f1ad00f1b40 msgr2=0x7f1ad00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- 172.21.15.112:0/2232304224 shutdown_connections 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- 172.21.15.112:0/2232304224 wait complete. 2023-11-16T14:04:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 Processor -- start 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- start start 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.511+0000 7f1acffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad0069f00 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1aceffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1aceffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38290/0 (socket says 172.21.15.112:38290) 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1aceffd700 1 -- 172.21.15.112:0/217314791 learned_addr learned my addr 172.21.15.112:0/217314791 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1aceffd700 1 -- 172.21.15.112:0/217314791 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4005040 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1aceffd700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1ac40089f0 tx=0x7f1ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac4009320 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad006a0a0 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad006a530 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1ac4013ec0 con 0x7f1ad00f63e0 2023-11-16T14:04:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.512+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac4014d00 con 0x7f1ad00f63e0 2023-11-16T14:04:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.513+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1ac4014420 con 0x7f1ad00f63e0 2023-11-16T14:04:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.513+0000 7f1ab77fe700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 0x7f1ab8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.513+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1ac40424c0 con 0x7f1ad00f63e0 2023-11-16T14:04:21.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.513+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad00591e0 con 0x7f1ad00f63e0 2023-11-16T14:04:21.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.513+0000 7f1ace7fc700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 0x7f1ab8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:21.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.514+0000 7f1ace7fc700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 0x7f1ab8033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1ac00096a0 tx=0x7f1ac0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:21.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.518+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1ac4014e60 con 0x7f1ad00f63e0 2023-11-16T14:04:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.725+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ad006ac30 con 0x7f1ad00f63e0 2023-11-16T14:04:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.726+0000 7f1ab77fe700 1 -- 172.21.15.112:0/217314791 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ac402b020 con 0x7f1ad00f63e0 2023-11-16T14:04:21.728 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:21.728 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:21.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 msgr2=0x7f1ab8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 0x7f1ab8033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1ac00096a0 tx=0x7f1ac0006b40).stop 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 msgr2=0x7f1ad006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1ac40089f0 tx=0x7f1ac400b040).stop 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 shutdown_connections 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1ab8031090 0x7f1ab8033550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 --2- 172.21.15.112:0/217314791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad00f63e0 0x7f1ad006ba80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 >> 172.21.15.112:0/217314791 conn(0x7f1ad00f1b40 msgr2=0x7f1ad00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 shutdown_connections 2023-11-16T14:04:21.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:21.728+0000 7f1acffff700 1 -- 172.21.15.112:0/217314791 wait complete. 2023-11-16T14:04:21.731 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:20.862589+0000 mgr.smithi061.zcjido (mgr.14156) 85 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:21.726586+0000 mon.smithi061 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.112:0/217314791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:23.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:23.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:22.862892+0000 mgr.smithi061.zcjido (mgr.14156) 86 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:24.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- 172.21.15.112:0/406557855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 msgr2=0x7f3e4c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:24.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 --2- 172.21.15.112:0/406557855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c0f69b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f3e3c004760 tx=0x7f3e3c009ad0).stop 2023-11-16T14:04:24.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- 172.21.15.112:0/406557855 shutdown_connections 2023-11-16T14:04:24.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 --2- 172.21.15.112:0/406557855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c0f69b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- 172.21.15.112:0/406557855 >> 172.21.15.112:0/406557855 conn(0x7f3e4c0f1cd0 msgr2=0x7f3e4c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- 172.21.15.112:0/406557855 shutdown_connections 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- 172.21.15.112:0/406557855 wait complete. 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 Processor -- start 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.166+0000 7f3e5296b700 1 -- start start 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e5296b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:24.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e5296b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c106cb0 con 0x7f3e4c0f6590 2023-11-16T14:04:24.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38304/0 (socket says 172.21.15.112:38304) 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e4bfff700 1 -- 172.21.15.112:0/992445349 learned_addr learned my addr 172.21.15.112:0/992445349 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e4bfff700 1 -- 172.21.15.112:0/992445349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c009d70 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e4bfff700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3e3c004170 tx=0x7f3e3c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.167+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e3c00e040 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c106eb0 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3e3c008ef0 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e3c014c90 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c107340 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3e3c014420 con 0x7f3e4c0f6590 2023-11-16T14:04:24.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.168+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4c045bb0 con 0x7f3e4c0f6590 2023-11-16T14:04:24.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.169+0000 7f3e48ff9700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 0x7f3e34033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:24.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.169+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3e3c0422d0 con 0x7f3e4c0f6590 2023-11-16T14:04:24.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.169+0000 7f3e4b7fe700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 0x7f3e34033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:24.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.169+0000 7f3e4b7fe700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 0x7f3e34033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3e400096a0 tx=0x7f3e40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:24.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.173+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3e3c026300 con 0x7f3e4c0f6590 2023-11-16T14:04:24.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.381+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e4c108190 con 0x7f3e4c0f6590 2023-11-16T14:04:24.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.382+0000 7f3e48ff9700 1 -- 172.21.15.112:0/992445349 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e3c012b20 con 0x7f3e4c0f6590 2023-11-16T14:04:24.384 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:24.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 msgr2=0x7f3e34033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 0x7f3e34033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3e400096a0 tx=0x7f3e40006b40).stop 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 msgr2=0x7f3e4c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3e3c004170 tx=0x7f3e3c009ad0).stop 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 shutdown_connections 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3e34031090 0x7f3e34033550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 --2- 172.21.15.112:0/992445349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e4c0f6590 0x7f3e4c1067c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 >> 172.21.15.112:0/992445349 conn(0x7f3e4c0f1cd0 msgr2=0x7f3e4c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:24.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 shutdown_connections 2023-11-16T14:04:24.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:24.384+0000 7f3e5296b700 1 -- 172.21.15.112:0/992445349 wait complete. 2023-11-16T14:04:24.387 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:25.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:24.382336+0000 mon.smithi061 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.112:0/992445349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:25.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:25.788 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:26.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:24.863197+0000 mgr.smithi061.zcjido (mgr.14156) 87 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:26.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 -- 172.21.15.112:0/589775192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 msgr2=0x7fbd600f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:26.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 --2- 172.21.15.112:0/589775192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd600f69b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fbd54004760 tx=0x7fbd54009ad0).stop 2023-11-16T14:04:26.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 -- 172.21.15.112:0/589775192 shutdown_connections 2023-11-16T14:04:26.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 --2- 172.21.15.112:0/589775192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd600f69b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:26.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 -- 172.21.15.112:0/589775192 >> 172.21.15.112:0/589775192 conn(0x7fbd600f1cd0 msgr2=0x7fbd600f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 -- 172.21.15.112:0/589775192 shutdown_connections 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.835+0000 7fbd670c1700 1 -- 172.21.15.112:0/589775192 wait complete. 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd670c1700 1 Processor -- start 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd670c1700 1 -- start start 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd670c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd670c1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60106cb0 con 0x7fbd600f6590 2023-11-16T14:04:26.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd64e5d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:26.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd64e5d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:37606/0 (socket says 172.21.15.112:37606) 2023-11-16T14:04:26.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.836+0000 7fbd64e5d700 1 -- 172.21.15.112:0/372265619 learned_addr learned my addr 172.21.15.112:0/372265619 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:26.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd64e5d700 1 -- 172.21.15.112:0/372265619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd54009d70 con 0x7fbd600f6590 2023-11-16T14:04:26.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd64e5d700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fbd54004170 tx=0x7fbd54009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:26.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5400e040 con 0x7fbd600f6590 2023-11-16T14:04:26.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60106eb0 con 0x7fbd600f6590 2023-11-16T14:04:26.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd54008ef0 con 0x7fbd600f6590 2023-11-16T14:04:26.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd54014c90 con 0x7fbd600f6590 2023-11-16T14:04:26.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.837+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd60107340 con 0x7fbd600f6590 2023-11-16T14:04:26.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.838+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbd54014420 con 0x7fbd600f6590 2023-11-16T14:04:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.838+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd60045bb0 con 0x7fbd600f6590 2023-11-16T14:04:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.838+0000 7fbd5d7fa700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 0x7fbd50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.838+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd540421f0 con 0x7fbd600f6590 2023-11-16T14:04:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.838+0000 7fbd5ffff700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 0x7fbd50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.839+0000 7fbd5ffff700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 0x7fbd50033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbd4c0096a0 tx=0x7fbd4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:26.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:26.843+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd54026300 con 0x7fbd600f6590 2023-11-16T14:04:27.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.055+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd601081f0 con 0x7fbd600f6590 2023-11-16T14:04:27.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.055+0000 7fbd5d7fa700 1 -- 172.21.15.112:0/372265619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd54012b20 con 0x7fbd600f6590 2023-11-16T14:04:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:27.058 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:27.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 msgr2=0x7fbd50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:27.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 0x7fbd50033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbd4c0096a0 tx=0x7fbd4c006b40).stop 2023-11-16T14:04:27.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 msgr2=0x7fbd601067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:27.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fbd54004170 tx=0x7fbd54009ad0).stop 2023-11-16T14:04:27.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 shutdown_connections 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd50031090 0x7fbd50033550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 --2- 172.21.15.112:0/372265619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd600f6590 0x7fbd601067c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 >> 172.21.15.112:0/372265619 conn(0x7fbd600f1cd0 msgr2=0x7fbd600f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 shutdown_connections 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:27.057+0000 7fbd670c1700 1 -- 172.21.15.112:0/372265619 wait complete. 2023-11-16T14:04:27.062 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:26.863434+0000 mgr.smithi061.zcjido (mgr.14156) 88 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:27.055918+0000 mon.smithi061 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.112:0/372265619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:28.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:28.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:29.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 -- 172.21.15.112:0/2274469632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 msgr2=0x7fe16c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 --2- 172.21.15.112:0/2274469632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c0f58d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe15c009fb0 tx=0x7fe15c009b30).stop 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 -- 172.21.15.112:0/2274469632 shutdown_connections 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 --2- 172.21.15.112:0/2274469632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c0f58d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 -- 172.21.15.112:0/2274469632 >> 172.21.15.112:0/2274469632 conn(0x7fe16c0f0c10 msgr2=0x7fe16c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 -- 172.21.15.112:0/2274469632 shutdown_connections 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.561+0000 7fe1744cf700 1 -- 172.21.15.112:0/2274469632 wait complete. 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe1744cf700 1 Processor -- start 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe1744cf700 1 -- start start 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe1744cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe1744cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c060320 con 0x7fe16c0f54b0 2023-11-16T14:04:29.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe17226b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe17226b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:37620/0 (socket says 172.21.15.112:37620) 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe17226b700 1 -- 172.21.15.112:0/1666659535 learned_addr learned my addr 172.21.15.112:0/1666659535 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.562+0000 7fe17226b700 1 -- 172.21.15.112:0/1666659535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c005040 con 0x7fe16c0f54b0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe17226b700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe15c0089f0 tx=0x7fe15c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe15c009320 con 0x7fe16c0f54b0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c060520 con 0x7fe16c0f54b0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe15c013ec0 con 0x7fe16c0f54b0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe15c014d30 con 0x7fe16c0f54b0 2023-11-16T14:04:29.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.563+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c109b50 con 0x7fe16c0f54b0 2023-11-16T14:04:29.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.564+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe15c0095f0 con 0x7fe16c0f54b0 2023-11-16T14:04:29.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.564+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c045bb0 con 0x7fe16c0f54b0 2023-11-16T14:04:29.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.564+0000 7fe162ffd700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 0x7fe158033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:29.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.564+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe15c0423e0 con 0x7fe16c0f54b0 2023-11-16T14:04:29.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.564+0000 7fe171a6a700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 0x7fe158033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:29.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.565+0000 7fe171a6a700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 0x7fe158033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe1680096a0 tx=0x7fe168006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:29.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.569+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe15c01d980 con 0x7fe16c0f54b0 2023-11-16T14:04:29.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.780+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe16c10a370 con 0x7fe16c0f54b0 2023-11-16T14:04:29.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.780+0000 7fe162ffd700 1 -- 172.21.15.112:0/1666659535 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe15c01d380 con 0x7fe16c0f54b0 2023-11-16T14:04:29.782 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:29.783 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 msgr2=0x7fe158033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 0x7fe158033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe1680096a0 tx=0x7fe168006b40).stop 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 msgr2=0x7fe16c1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe15c0089f0 tx=0x7fe15c00b040).stop 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 shutdown_connections 2023-11-16T14:04:29.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe158031090 0x7fe158033550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:29.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 --2- 172.21.15.112:0/1666659535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe16c0f54b0 0x7fe16c1094a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:29.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 >> 172.21.15.112:0/1666659535 conn(0x7fe16c0f0c10 msgr2=0x7fe16c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:29.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 shutdown_connections 2023-11-16T14:04:29.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:29.782+0000 7fe1744cf700 1 -- 172.21.15.112:0/1666659535 wait complete. 2023-11-16T14:04:29.785 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:28.863701+0000 mgr.smithi061.zcjido (mgr.14156) 89 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:29.780871+0000 mon.smithi061 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.112:0/1666659535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:31.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:31.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:32.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 -- 172.21.15.112:0/3603719960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 msgr2=0x7f19900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:32.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 --2- 172.21.15.112:0/3603719960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19900f58a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f198c009fb0 tx=0x7f198c009b30).stop 2023-11-16T14:04:32.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 -- 172.21.15.112:0/3603719960 shutdown_connections 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 --2- 172.21.15.112:0/3603719960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19900f58a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 -- 172.21.15.112:0/3603719960 >> 172.21.15.112:0/3603719960 conn(0x7f19900f0be0 msgr2=0x7f19900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 -- 172.21.15.112:0/3603719960 shutdown_connections 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.246+0000 7f1998524700 1 -- 172.21.15.112:0/3603719960 wait complete. 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1998524700 1 Processor -- start 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1998524700 1 -- start start 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1998524700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1998524700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1990105be0 con 0x7f19900f5480 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f19962c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f19962c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:37634/0 (socket says 172.21.15.112:37634) 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f19962c0700 1 -- 172.21.15.112:0/2650327829 learned_addr learned my addr 172.21.15.112:0/2650327829 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:32.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f19962c0700 1 -- 172.21.15.112:0/2650327829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c005040 con 0x7f19900f5480 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f19962c0700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f198c0089f0 tx=0x7f198c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c009320 con 0x7f19900f5480 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.247+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1990105de0 con 0x7f19900f5480 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f198c013ec0 con 0x7f19900f5480 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c014d30 con 0x7f19900f5480 2023-11-16T14:04:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1990106270 con 0x7f19900f5480 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f198c0095f0 con 0x7f19900f5480 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1986ffd700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 0x7f197c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.248+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f198c0424a0 con 0x7f19900f5480 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.249+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1990045bb0 con 0x7f19900f5480 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.249+0000 7f1995abf700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 0x7f197c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.249+0000 7f1995abf700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 0x7f197c033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f19800096a0 tx=0x7f1980006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:32.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.253+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f198c01d7f0 con 0x7f19900f5480 2023-11-16T14:04:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:30.863979+0000 mgr.smithi061.zcjido (mgr.14156) 90 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:32.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.464+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1990106f20 con 0x7f19900f5480 2023-11-16T14:04:32.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.464+0000 7f1986ffd700 1 -- 172.21.15.112:0/2650327829 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f198c01d7f0 con 0x7f19900f5480 2023-11-16T14:04:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:32.467 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 msgr2=0x7f197c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 0x7f197c033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f19800096a0 tx=0x7f1980006b40).stop 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 msgr2=0x7f19901056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f198c0089f0 tx=0x7f198c00b040).stop 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 shutdown_connections 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f197c031090 0x7f197c033550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 --2- 172.21.15.112:0/2650327829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19900f5480 0x7f19901056f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 >> 172.21.15.112:0/2650327829 conn(0x7f19900f0be0 msgr2=0x7f19900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:32.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 shutdown_connections 2023-11-16T14:04:32.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:32.466+0000 7f1998524700 1 -- 172.21.15.112:0/2650327829 wait complete. 2023-11-16T14:04:32.470 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:33.259 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:32 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:32.465865+0000 mon.smithi061 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.112:0/2650327829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:33.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:33.823 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:34.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:32.864244+0000 mgr.smithi061.zcjido (mgr.14156) 91 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:34.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 -- 172.21.15.112:0/3454933931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 msgr2=0x7fa6ec0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:34.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3454933931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec0f58d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa6dc009fb0 tx=0x7fa6dc009b30).stop 2023-11-16T14:04:34.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 -- 172.21.15.112:0/3454933931 shutdown_connections 2023-11-16T14:04:34.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3454933931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec0f58d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:34.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 -- 172.21.15.112:0/3454933931 >> 172.21.15.112:0/3454933931 conn(0x7fa6ec0f0c10 msgr2=0x7fa6ec0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:34.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 -- 172.21.15.112:0/3454933931 shutdown_connections 2023-11-16T14:04:34.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.879+0000 7fa6f284e700 1 -- 172.21.15.112:0/3454933931 wait complete. 2023-11-16T14:04:34.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.880+0000 7fa6f284e700 1 Processor -- start 2023-11-16T14:04:34.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.880+0000 7fa6f284e700 1 -- start start 2023-11-16T14:04:34.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.880+0000 7fa6f284e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:34.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.880+0000 7fa6f284e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec105b90 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:37656/0 (socket says 172.21.15.112:37656) 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6ebfff700 1 -- 172.21.15.112:0/3831431136 learned_addr learned my addr 172.21.15.112:0/3831431136 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6ebfff700 1 -- 172.21.15.112:0/3831431136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc005040 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6ebfff700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa6dc0089f0 tx=0x7fa6dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6dc009320 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec105d90 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa6dc013ec0 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6dc014d30 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.881+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec106220 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.882+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa6dc0095f0 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.882+0000 7fa6e8ff9700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 0x7fa6d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.882+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa6dc0424a0 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.882+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec045bb0 con 0x7fa6ec0f54b0 2023-11-16T14:04:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.882+0000 7fa6eb7fe700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 0x7fa6d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.883+0000 7fa6eb7fe700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 0x7fa6d4033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa6e00096a0 tx=0x7fa6e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:34.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:34.887+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa6dc01d7f0 con 0x7fa6ec0f54b0 2023-11-16T14:04:35.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.095+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6ec0fb090 con 0x7fa6ec0f54b0 2023-11-16T14:04:35.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.095+0000 7fa6e8ff9700 1 -- 172.21.15.112:0/3831431136 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6dc01d7f0 con 0x7fa6ec0f54b0 2023-11-16T14:04:35.098 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:35.098 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 msgr2=0x7fa6d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 0x7fa6d4033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa6e00096a0 tx=0x7fa6e0006b40).stop 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 msgr2=0x7fa6ec1056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa6dc0089f0 tx=0x7fa6dc00b040).stop 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 shutdown_connections 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa6d4031090 0x7fa6d4033550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 --2- 172.21.15.112:0/3831431136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ec0f54b0 0x7fa6ec1056a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 >> 172.21.15.112:0/3831431136 conn(0x7fa6ec0f0c10 msgr2=0x7fa6ec0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:35.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 shutdown_connections 2023-11-16T14:04:35.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:35.097+0000 7fa6f284e700 1 -- 172.21.15.112:0/3831431136 wait complete. 2023-11-16T14:04:35.101 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:34.864527+0000 mgr.smithi061.zcjido (mgr.14156) 92 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:34.980100+0000 mon.smithi061 (mon.0) 169 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:04:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:34.980884+0000 mon.smithi061 (mon.0) 170 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:04:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:35.096714+0000 mon.smithi061 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.112:0/3831431136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:36.508 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:36.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:37.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 -- 172.21.15.112:0/3842363973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 msgr2=0x7f50800fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:37.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 --2- 172.21.15.112:0/3842363973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f50800fb5e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5074009fb0 tx=0x7f5074009b30).stop 2023-11-16T14:04:37.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 -- 172.21.15.112:0/3842363973 shutdown_connections 2023-11-16T14:04:37.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 --2- 172.21.15.112:0/3842363973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f50800fb5e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:37.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 -- 172.21.15.112:0/3842363973 >> 172.21.15.112:0/3842363973 conn(0x7f50800f0be0 msgr2=0x7f50800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:37.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 -- 172.21.15.112:0/3842363973 shutdown_connections 2023-11-16T14:04:37.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.570+0000 7f508522a700 1 -- 172.21.15.112:0/3842363973 wait complete. 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f508522a700 1 Processor -- start 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f508522a700 1 -- start start 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f508522a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f508522a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5080108e60 con 0x7f50800602e0 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f507ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f507ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57428/0 (socket says 172.21.15.112:57428) 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f507ed9d700 1 -- 172.21.15.112:0/3264208857 learned_addr learned my addr 172.21.15.112:0/3264208857 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f507ed9d700 1 -- 172.21.15.112:0/3264208857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074005040 con 0x7f50800602e0 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.571+0000 7f507ed9d700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5074009f80 tx=0x7f507400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5074009320 con 0x7f50800602e0 2023-11-16T14:04:37.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5080109060 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5074013680 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f507401eda0 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50801094f0 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.572+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f507401e420 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.573+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5080045bb0 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.573+0000 7f50677fe700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 0x7f5068033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.573+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5074042420 con 0x7f50800602e0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.573+0000 7f507e59c700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 0x7f5068033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.573+0000 7f507e59c700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 0x7f5068033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f50700096a0 tx=0x7f5070006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:37.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.578+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f507401d380 con 0x7f50800602e0 2023-11-16T14:04:37.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.785+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f508010a340 con 0x7f50800602e0 2023-11-16T14:04:37.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.786+0000 7f50677fe700 1 -- 172.21.15.112:0/3264208857 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f507401cb10 con 0x7f50800602e0 2023-11-16T14:04:37.789 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:37.789 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 msgr2=0x7f5068033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 0x7f5068033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f50700096a0 tx=0x7f5070006b40).stop 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 msgr2=0x7f5080108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5074009f80 tx=0x7f507400b040).stop 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 shutdown_connections 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5068031090 0x7f5068033550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:37.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 --2- 172.21.15.112:0/3264208857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f50800602e0 0x7f5080108970 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:37.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 >> 172.21.15.112:0/3264208857 conn(0x7f50800f0be0 msgr2=0x7f50800f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:37.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 shutdown_connections 2023-11-16T14:04:37.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:37.787+0000 7f508522a700 1 -- 172.21.15.112:0/3264208857 wait complete. 2023-11-16T14:04:37.791 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:36.864921+0000 mgr.smithi061.zcjido (mgr.14156) 93 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:37.787175+0000 mon.smithi061 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.112:0/3264208857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:39.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:39.144 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:38.865263+0000 mgr.smithi061.zcjido (mgr.14156) 94 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- 172.21.15.112:0/3422286773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 msgr2=0x7f6ff80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/3422286773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff80f69b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6fe8004760 tx=0x7f6fe8009ad0).stop 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- 172.21.15.112:0/3422286773 shutdown_connections 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/3422286773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff80f69b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- 172.21.15.112:0/3422286773 >> 172.21.15.112:0/3422286773 conn(0x7f6ff80f1cd0 msgr2=0x7f6ff80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- 172.21.15.112:0/3422286773 shutdown_connections 2023-11-16T14:04:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- 172.21.15.112:0/3422286773 wait complete. 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 Processor -- start 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- start start 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.333+0000 7f6fffcd9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8106cb0 con 0x7f6ff80f6590 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6ffda75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6ffda75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57448/0 (socket says 172.21.15.112:57448) 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6ffda75700 1 -- 172.21.15.112:0/1541051386 learned_addr learned my addr 172.21.15.112:0/1541051386 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6ffda75700 1 -- 172.21.15.112:0/1541051386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009d70 con 0x7f6ff80f6590 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6ffda75700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f6fe8004170 tx=0x7f6fe8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe800e040 con 0x7f6ff80f6590 2023-11-16T14:04:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8106eb0 con 0x7f6ff80f6590 2023-11-16T14:04:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6fe8008ef0 con 0x7f6ff80f6590 2023-11-16T14:04:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe8014c90 con 0x7f6ff80f6590 2023-11-16T14:04:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.334+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8107340 con 0x7f6ff80f6590 2023-11-16T14:04:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.335+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6fe8014420 con 0x7f6ff80f6590 2023-11-16T14:04:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.335+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8045bb0 con 0x7f6ff80f6590 2023-11-16T14:04:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.335+0000 7f6fee7fc700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 0x7f6fe4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.335+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6fe8042260 con 0x7f6ff80f6590 2023-11-16T14:04:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.335+0000 7f6ffd274700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 0x7f6fe4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.336+0000 7f6ffd274700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 0x7f6fe4033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f6ff40096a0 tx=0x7f6ff4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.340+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6fe8026300 con 0x7f6ff80f6590 2023-11-16T14:04:40.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.550+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ff8108190 con 0x7f6ff80f6590 2023-11-16T14:04:40.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.551+0000 7f6fee7fc700 1 -- 172.21.15.112:0/1541051386 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fe8012b20 con 0x7f6ff80f6590 2023-11-16T14:04:40.554 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:40.554 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:40.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 msgr2=0x7f6fe4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:40.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 0x7f6fe4033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f6ff40096a0 tx=0x7f6ff4006b40).stop 2023-11-16T14:04:40.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 msgr2=0x7f6ff81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:40.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f6fe8004170 tx=0x7f6fe8009ad0).stop 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 shutdown_connections 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe4031090 0x7f6fe4033550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 --2- 172.21.15.112:0/1541051386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ff80f6590 0x7f6ff81067c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.552+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 >> 172.21.15.112:0/1541051386 conn(0x7f6ff80f1cd0 msgr2=0x7f6ff80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.553+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 shutdown_connections 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:40.553+0000 7f6fffcd9700 1 -- 172.21.15.112:0/1541051386 wait complete. 2023-11-16T14:04:40.556 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:41.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:40 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:40.552423+0000 mon.smithi061 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.112:0/1541051386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:41.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:41.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:40.865704+0000 mgr.smithi061.zcjido (mgr.14156) 95 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.974+0000 7f8a3b173700 1 -- 172.21.15.112:0/2589520470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 msgr2=0x7f8a340f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.974+0000 7f8a3b173700 1 --2- 172.21.15.112:0/2589520470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a340f56f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8a24009fb0 tx=0x7f8a24009b30).stop 2023-11-16T14:04:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.974+0000 7f8a3b173700 1 -- 172.21.15.112:0/2589520470 shutdown_connections 2023-11-16T14:04:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.974+0000 7f8a3b173700 1 --2- 172.21.15.112:0/2589520470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a340f56f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.974+0000 7f8a3b173700 1 -- 172.21.15.112:0/2589520470 >> 172.21.15.112:0/2589520470 conn(0x7f8a340f0a30 msgr2=0x7f8a340f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 -- 172.21.15.112:0/2589520470 shutdown_connections 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 -- 172.21.15.112:0/2589520470 wait complete. 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 Processor -- start 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 -- start start 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.975+0000 7f8a3b173700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34060320 con 0x7f8a340f52d0 2023-11-16T14:04:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3a171700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3a171700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57458/0 (socket says 172.21.15.112:57458) 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3a171700 1 -- 172.21.15.112:0/1617753842 learned_addr learned my addr 172.21.15.112:0/1617753842 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3a171700 1 -- 172.21.15.112:0/1617753842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24005040 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3a171700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8a240089f0 tx=0x7f8a2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a24009320 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a34060520 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8a24013ec0 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a24014d30 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.976+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a34109a20 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.977+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a34045bb0 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.977+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8a240095f0 con 0x7f8a340f52d0 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.978+0000 7f8a2affd700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 0x7f8a20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.978+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8a24042420 con 0x7f8a340f52d0 2023-11-16T14:04:42.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.978+0000 7f8a39970700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 0x7f8a20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:42.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.978+0000 7f8a39970700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 0x7f8a20033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8a300096a0 tx=0x7f8a30006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:42.983+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8a2401d7f0 con 0x7f8a340f52d0 2023-11-16T14:04:43.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.190+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a3410a230 con 0x7f8a340f52d0 2023-11-16T14:04:43.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.191+0000 7f8a2affd700 1 -- 172.21.15.112:0/1617753842 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a2400fe40 con 0x7f8a340f52d0 2023-11-16T14:04:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:43.194 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:43.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.192+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 msgr2=0x7f8a20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:43.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.192+0000 7f8a3b173700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 0x7f8a20033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8a300096a0 tx=0x7f8a30006b40).stop 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.192+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 msgr2=0x7f8a34109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.192+0000 7f8a3b173700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8a240089f0 tx=0x7f8a2400b040).stop 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 shutdown_connections 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8a20031090 0x7f8a20033550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 --2- 172.21.15.112:0/1617753842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a340f52d0 0x7f8a34109370 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 >> 172.21.15.112:0/1617753842 conn(0x7f8a340f0a30 msgr2=0x7f8a340f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 shutdown_connections 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:43.193+0000 7f8a3b173700 1 -- 172.21.15.112:0/1617753842 wait complete. 2023-11-16T14:04:43.196 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:42.866123+0000 mgr.smithi061.zcjido (mgr.14156) 96 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:43.192338+0000 mon.smithi061 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.112:0/1617753842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:44.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:44.582 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.680+0000 7fc26fb19700 1 -- 172.21.15.112:0/2169469798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 msgr2=0x7fc2680f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.680+0000 7fc26fb19700 1 --2- 172.21.15.112:0/2169469798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2680f58a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc258009fb0 tx=0x7fc258009b30).stop 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- 172.21.15.112:0/2169469798 shutdown_connections 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 --2- 172.21.15.112:0/2169469798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2680f58a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- 172.21.15.112:0/2169469798 >> 172.21.15.112:0/2169469798 conn(0x7fc2680f0be0 msgr2=0x7fc2680f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- 172.21.15.112:0/2169469798 shutdown_connections 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- 172.21.15.112:0/2169469798 wait complete. 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 Processor -- start 2023-11-16T14:04:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- start start 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.681+0000 7fc26fb19700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268105be0 con 0x7fc2680f5480 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26d8b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26d8b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34108/0 (socket says 172.21.15.112:34108) 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26d8b5700 1 -- 172.21.15.112:0/427291285 learned_addr learned my addr 172.21.15.112:0/427291285 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26d8b5700 1 -- 172.21.15.112:0/427291285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258005040 con 0x7fc2680f5480 2023-11-16T14:04:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26d8b5700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc2580089f0 tx=0x7fc25800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc258009320 con 0x7fc2680f5480 2023-11-16T14:04:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc268105de0 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc258013ec0 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc258014d30 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.682+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc268106270 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.683+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc2580095f0 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.683+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc268045bb0 con 0x7fc2680f5480 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.683+0000 7fc25e7fc700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 0x7fc2540335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.683+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc258042410 con 0x7fc2680f5480 2023-11-16T14:04:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.684+0000 7fc26d0b4700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 0x7fc2540335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.684+0000 7fc26d0b4700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 0x7fc2540335a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.688+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc25801d980 con 0x7fc2680f5480 2023-11-16T14:04:45.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.898+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2680faf70 con 0x7fc2680f5480 2023-11-16T14:04:45.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.898+0000 7fc25e7fc700 1 -- 172.21.15.112:0/427291285 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc25801d380 con 0x7fc2680f5480 2023-11-16T14:04:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:45.902 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:45.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 msgr2=0x7fc2540335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:45.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 0x7fc2540335a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).stop 2023-11-16T14:04:45.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 msgr2=0x7fc2681056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:45.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc2580089f0 tx=0x7fc25800b040).stop 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 shutdown_connections 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc2540310e0 0x7fc2540335a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 --2- 172.21.15.112:0/427291285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f5480 0x7fc2681056f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 >> 172.21.15.112:0/427291285 conn(0x7fc2680f0be0 msgr2=0x7fc2680f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 shutdown_connections 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:45.900+0000 7fc26fb19700 1 -- 172.21.15.112:0/427291285 wait complete. 2023-11-16T14:04:45.904 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:46.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:44.866462+0000 mgr.smithi061.zcjido (mgr.14156) 97 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:46.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:45.899893+0000 mon.smithi061 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.112:0/427291285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:47.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:47.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:46.866812+0000 mgr.smithi061.zcjido (mgr.14156) 98 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:48.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- 172.21.15.112:0/1576148915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 msgr2=0x7fc2680f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:48.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 --2- 172.21.15.112:0/1576148915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2680f69b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc258004760 tx=0x7fc258009ad0).stop 2023-11-16T14:04:48.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- 172.21.15.112:0/1576148915 shutdown_connections 2023-11-16T14:04:48.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 --2- 172.21.15.112:0/1576148915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2680f69b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- 172.21.15.112:0/1576148915 >> 172.21.15.112:0/1576148915 conn(0x7fc2680f1cd0 msgr2=0x7fc2680f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- 172.21.15.112:0/1576148915 shutdown_connections 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- 172.21.15.112:0/1576148915 wait complete. 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 Processor -- start 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.346+0000 7fc26f909700 1 -- start start 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26f909700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26f909700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268106cb0 con 0x7fc2680f6590 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26d6a5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26d6a5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34128/0 (socket says 172.21.15.112:34128) 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26d6a5700 1 -- 172.21.15.112:0/932584507 learned_addr learned my addr 172.21.15.112:0/932584507 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26d6a5700 1 -- 172.21.15.112:0/932584507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258009d70 con 0x7fc2680f6590 2023-11-16T14:04:48.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26d6a5700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc258004170 tx=0x7fc258009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:48.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc25800e040 con 0x7fc2680f6590 2023-11-16T14:04:48.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc258008ef0 con 0x7fc2680f6590 2023-11-16T14:04:48.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc268106eb0 con 0x7fc2680f6590 2023-11-16T14:04:48.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.347+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc258014c90 con 0x7fc2680f6590 2023-11-16T14:04:48.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.348+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc268107340 con 0x7fc2680f6590 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.348+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc258014420 con 0x7fc2680f6590 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.348+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc268045bb0 con 0x7fc2680f6590 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.349+0000 7fc25dffb700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 0x7fc254033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.349+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc2580422b0 con 0x7fc2680f6590 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.349+0000 7fc26cea4700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 0x7fc254033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:48.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.349+0000 7fc26cea4700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 0x7fc254033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:48.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.354+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc258026300 con 0x7fc2680f6590 2023-11-16T14:04:48.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.561+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2681081f0 con 0x7fc2680f6590 2023-11-16T14:04:48.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.563+0000 7fc25dffb700 1 -- 172.21.15.112:0/932584507 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc258012b20 con 0x7fc2680f6590 2023-11-16T14:04:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:48.565 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:48.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 msgr2=0x7fc254033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 0x7fc254033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).stop 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 msgr2=0x7fc2681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc258004170 tx=0x7fc258009ad0).stop 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 shutdown_connections 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc254031090 0x7fc254033550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 --2- 172.21.15.112:0/932584507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2680f6590 0x7fc2681067c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 >> 172.21.15.112:0/932584507 conn(0x7fc2680f1cd0 msgr2=0x7fc2680f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 shutdown_connections 2023-11-16T14:04:48.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:48.564+0000 7fc26f909700 1 -- 172.21.15.112:0/932584507 wait complete. 2023-11-16T14:04:48.568 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:49.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:48.563421+0000 mon.smithi061 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.112:0/932584507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:49.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:49.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:48.867080+0000 mgr.smithi061.zcjido (mgr.14156) 99 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- 172.21.15.112:0/3608617358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 msgr2=0x7f812c0f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 --2- 172.21.15.112:0/3608617358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c0f9200 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8120009fb0 tx=0x7f8120009b30).stop 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- 172.21.15.112:0/3608617358 shutdown_connections 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 --2- 172.21.15.112:0/3608617358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c0f9200 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- 172.21.15.112:0/3608617358 >> 172.21.15.112:0/3608617358 conn(0x7f812c0f0be0 msgr2=0x7f812c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:50.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- 172.21.15.112:0/3608617358 shutdown_connections 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- 172.21.15.112:0/3608617358 wait complete. 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 Processor -- start 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- start start 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.990+0000 7f8130b97700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c106c20 con 0x7f812c0602c0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f812a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f812a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34140/0 (socket says 172.21.15.112:34140) 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f812a59c700 1 -- 172.21.15.112:0/913404227 learned_addr learned my addr 172.21.15.112:0/913404227 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f812a59c700 1 -- 172.21.15.112:0/913404227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120005040 con 0x7f812c0602c0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f812a59c700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8120013fd0 tx=0x7f8120009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8120018070 con 0x7f812c0602c0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c106e20 con 0x7f812c0602c0 2023-11-16T14:04:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8120007b10 con 0x7f812c0602c0 2023-11-16T14:04:50.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812001c3c0 con 0x7f812c0602c0 2023-11-16T14:04:50.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.991+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c1072b0 con 0x7f812c0602c0 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.992+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8120007c80 con 0x7f812c0602c0 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.992+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812c045bb0 con 0x7f812c0602c0 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.992+0000 7f811affd700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 0x7f81140335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.992+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8120042940 con 0x7f812c0602c0 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.993+0000 7f8129d9b700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 0x7f81140335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:50.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.993+0000 7f8129d9b700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 0x7f81140335a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f811c0096a0 tx=0x7f811c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:51.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:50.997+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f812001cb10 con 0x7f812c0602c0 2023-11-16T14:04:51.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.208+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f812c108160 con 0x7f812c0602c0 2023-11-16T14:04:51.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.209+0000 7f811affd700 1 -- 172.21.15.112:0/913404227 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8120025b40 con 0x7f812c0602c0 2023-11-16T14:04:51.212 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:51.212 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 msgr2=0x7f81140335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 0x7f81140335a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f811c0096a0 tx=0x7f811c006b40).stop 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 msgr2=0x7f812c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8120013fd0 tx=0x7f8120009b30).stop 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 shutdown_connections 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81140310e0 0x7f81140335a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 --2- 172.21.15.112:0/913404227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f812c0602c0 0x7f812c106730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 >> 172.21.15.112:0/913404227 conn(0x7f812c0f0be0 msgr2=0x7f812c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 shutdown_connections 2023-11-16T14:04:51.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:51.211+0000 7f8130b97700 1 -- 172.21.15.112:0/913404227 wait complete. 2023-11-16T14:04:51.215 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:50.867290+0000 mgr.smithi061.zcjido (mgr.14156) 100 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:51.210547+0000 mon.smithi061 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.112:0/913404227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:52.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:52.598 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.680+0000 7ffb62165700 1 -- 172.21.15.112:0/3695082183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f5a50 msgr2=0x7ffb5c0f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.680+0000 7ffb62165700 1 --2- 172.21.15.112:0/3695082183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f5a50 0x7ffb5c0f5e70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ffb4c009fb0 tx=0x7ffb4c009b30).stop 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.680+0000 7ffb62165700 1 -- 172.21.15.112:0/3695082183 shutdown_connections 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.680+0000 7ffb62165700 1 --2- 172.21.15.112:0/3695082183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f5a50 0x7ffb5c0f5e70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.680+0000 7ffb62165700 1 -- 172.21.15.112:0/3695082183 >> 172.21.15.112:0/3695082183 conn(0x7ffb5c0f0360 msgr2=0x7ffb5c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 -- 172.21.15.112:0/3695082183 shutdown_connections 2023-11-16T14:04:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 -- 172.21.15.112:0/3695082183 wait complete. 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 Processor -- start 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 -- start start 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb62165700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c107460 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.681+0000 7ffb5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34154/0 (socket says 172.21.15.112:34154) 2023-11-16T14:04:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb5b7fe700 1 -- 172.21.15.112:0/2959230659 learned_addr learned my addr 172.21.15.112:0/2959230659 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb5b7fe700 1 -- 172.21.15.112:0/2959230659 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c005040 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb5b7fe700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7ffb4c009880 tx=0x7ffb4c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb4c013680 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ffb4c0095f0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c10bf60 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb4c014e40 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.682+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c10c2b0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.683+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ffb4c014420 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.683+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c045bb0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.683+0000 7ffb43fff700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 0x7ffb44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:53.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.683+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ffb4c0424f0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.683+0000 7ffb5affd700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 0x7ffb44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:53.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.684+0000 7ffb5affd700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 0x7ffb44033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ffb500096a0 tx=0x7ffb50006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.688+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ffb4c00e2a0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.896+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb5c10cb60 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.897+0000 7ffb43fff700 1 -- 172.21.15.112:0/2959230659 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb4c025ae0 con 0x7ffb5c0f53d0 2023-11-16T14:04:53.899 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:53.899 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 msgr2=0x7ffb44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 0x7ffb44033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ffb500096a0 tx=0x7ffb50006b40).stop 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 msgr2=0x7ffb5c10ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7ffb4c009880 tx=0x7ffb4c009b30).stop 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 shutdown_connections 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ffb44031090 0x7ffb44033550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 --2- 172.21.15.112:0/2959230659 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb5c0f53d0 0x7ffb5c10ba70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 >> 172.21.15.112:0/2959230659 conn(0x7ffb5c0f0360 msgr2=0x7ffb5c0f0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 shutdown_connections 2023-11-16T14:04:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:53.899+0000 7ffb62165700 1 -- 172.21.15.112:0/2959230659 wait complete. 2023-11-16T14:04:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:52.867629+0000 mgr.smithi061.zcjido (mgr.14156) 101 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:55.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:54 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:53.897697+0000 mon.smithi061 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.112:0/2959230659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:55.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:55.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:56.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:54.868003+0000 mgr.smithi061.zcjido (mgr.14156) 102 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:56.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2753473794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 msgr2=0x7f8fe40f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2753473794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe40f58d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8fcc009fb0 tx=0x7f8fcc009b30).stop 2023-11-16T14:04:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2753473794 shutdown_connections 2023-11-16T14:04:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2753473794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe40f58d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2753473794 >> 172.21.15.112:0/2753473794 conn(0x7f8fe40f0c10 msgr2=0x7f8fe40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2753473794 shutdown_connections 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.397+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2753473794 wait complete. 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe96c1700 1 Processor -- start 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe96c1700 1 -- start start 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe96c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe96c1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4106ca0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58308/0 (socket says 172.21.15.112:58308) 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.398+0000 7f8fe2ffd700 1 -- 172.21.15.112:0/2011914543 learned_addr learned my addr 172.21.15.112:0/2011914543 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fe2ffd700 1 -- 172.21.15.112:0/2011914543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc005040 con 0x7f8fe40f54b0 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fe2ffd700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f8fcc000c00 tx=0x7f8fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fcc0095f0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe4106ea0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8fcc007e50 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fcc014e70 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.399+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe4107330 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.400+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8fcc025070 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.400+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4045bb0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.400+0000 7f8fdb7fe700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 0x7f8fd00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.400+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8fcc041de0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.400+0000 7f8fe27fc700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 0x7f8fd00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:56.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.401+0000 7f8fe27fc700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 0x7f8fd00335a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8fd40096a0 tx=0x7f8fd4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:56.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.405+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8fcc00f8d0 con 0x7f8fe40f54b0 2023-11-16T14:04:56.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.613+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fe4108140 con 0x7f8fe40f54b0 2023-11-16T14:04:56.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.613+0000 7f8fdb7fe700 1 -- 172.21.15.112:0/2011914543 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fcc00f300 con 0x7f8fe40f54b0 2023-11-16T14:04:56.616 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:56.616 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 msgr2=0x7f8fd00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 0x7f8fd00335a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8fd40096a0 tx=0x7f8fd4006b40).stop 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 msgr2=0x7f8fe41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f8fcc000c00 tx=0x7f8fcc00b040).stop 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 shutdown_connections 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8fd00310e0 0x7f8fd00335a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 --2- 172.21.15.112:0/2011914543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe40f54b0 0x7f8fe41067b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 >> 172.21.15.112:0/2011914543 conn(0x7f8fe40f0c10 msgr2=0x7f8fe40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:56.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 shutdown_connections 2023-11-16T14:04:56.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:56.615+0000 7f8fe96c1700 1 -- 172.21.15.112:0/2011914543 wait complete. 2023-11-16T14:04:56.618 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:04:57.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:56 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:56.613883+0000 mon.smithi061 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.112:0/2011914543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:04:57.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:04:57.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:04:58.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:56.868360+0000 mgr.smithi061.zcjido (mgr.14156) 103 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:04:59.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 -- 172.21.15.112:0/2157198541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 msgr2=0x7fc3440f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:59.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 --2- 172.21.15.112:0/2157198541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3440f58a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc338009fb0 tx=0x7fc338009b30).stop 2023-11-16T14:04:59.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 -- 172.21.15.112:0/2157198541 shutdown_connections 2023-11-16T14:04:59.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 --2- 172.21.15.112:0/2157198541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3440f58a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:59.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 -- 172.21.15.112:0/2157198541 >> 172.21.15.112:0/2157198541 conn(0x7fc3440f0be0 msgr2=0x7fc3440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:59.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 -- 172.21.15.112:0/2157198541 shutdown_connections 2023-11-16T14:04:59.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 -- 172.21.15.112:0/2157198541 wait complete. 2023-11-16T14:04:59.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.047+0000 7fc349c36700 1 Processor -- start 2023-11-16T14:04:59.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc349c36700 1 -- start start 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc349c36700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc349c36700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc344108ec0 con 0x7fc3440f5480 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc3437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc3437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58336/0 (socket says 172.21.15.112:58336) 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc3437fe700 1 -- 172.21.15.112:0/2313545698 learned_addr learned my addr 172.21.15.112:0/2313545698 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:04:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc3437fe700 1 -- 172.21.15.112:0/2313545698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338005040 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.048+0000 7fc3437fe700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fc3380089f0 tx=0x7fc33800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.049+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc338009320 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.049+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3441090c0 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.049+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc338013ec0 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.049+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc338014d30 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.049+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc344109550 con 0x7fc3440f5480 2023-11-16T14:04:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc3380095f0 con 0x7fc3440f5480 2023-11-16T14:04:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc344045bb0 con 0x7fc3440f5480 2023-11-16T14:04:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc32bfff700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 0x7fc32c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:04:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc3380423e0 con 0x7fc3440f5480 2023-11-16T14:04:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc342ffd700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 0x7fc32c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:04:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.050+0000 7fc342ffd700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 0x7fc32c033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc3340096a0 tx=0x7fc334006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:04:59.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.055+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc33801d7f0 con 0x7fc3440f5480 2023-11-16T14:04:59.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.264+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc34410a400 con 0x7fc3440f5480 2023-11-16T14:04:59.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.265+0000 7fc32bfff700 1 -- 172.21.15.112:0/2313545698 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc33801d7f0 con 0x7fc3440f5480 2023-11-16T14:04:59.267 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:04:59.267 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:04:59.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 msgr2=0x7fc32c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 0x7fc32c033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc3340096a0 tx=0x7fc334006b40).stop 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 msgr2=0x7fc3441089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fc3380089f0 tx=0x7fc33800b040).stop 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 shutdown_connections 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc32c031090 0x7fc32c033550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 --2- 172.21.15.112:0/2313545698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3440f5480 0x7fc3441089d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 >> 172.21.15.112:0/2313545698 conn(0x7fc3440f0be0 msgr2=0x7fc3440f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 shutdown_connections 2023-11-16T14:04:59.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:04:59.267+0000 7fc349c36700 1 -- 172.21.15.112:0/2313545698 wait complete. 2023-11-16T14:04:59.270 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:04:58.868709+0000 mgr.smithi061.zcjido (mgr.14156) 104 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:04:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:04:59.265588+0000 mon.smithi061 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.112:0/2313545698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:00.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:00.633 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 -- 172.21.15.112:0/903285171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 msgr2=0x7fae200f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 --2- 172.21.15.112:0/903285171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae200f56f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fae1c009fb0 tx=0x7fae1c009b30).stop 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 -- 172.21.15.112:0/903285171 shutdown_connections 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 --2- 172.21.15.112:0/903285171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae200f56f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 -- 172.21.15.112:0/903285171 >> 172.21.15.112:0/903285171 conn(0x7fae200f0a30 msgr2=0x7fae200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 -- 172.21.15.112:0/903285171 shutdown_connections 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 -- 172.21.15.112:0/903285171 wait complete. 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.698+0000 7fae2663a700 1 Processor -- start 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae2663a700 1 -- start start 2023-11-16T14:05:01.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae2663a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:01.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae2663a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20108cf0 con 0x7fae200f52d0 2023-11-16T14:05:01.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae25638700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:01.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae25638700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58362/0 (socket says 172.21.15.112:58362) 2023-11-16T14:05:01.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae25638700 1 -- 172.21.15.112:0/1182720522 learned_addr learned my addr 172.21.15.112:0/1182720522 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:01.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae25638700 1 -- 172.21.15.112:0/1182720522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c005040 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.699+0000 7fae25638700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fae1c0089f0 tx=0x7fae1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.700+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c009320 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.700+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae20108ef0 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.700+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fae1c013ec0 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.700+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c014d30 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.700+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae20109380 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.701+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fae1c0095f0 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.701+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae20045bb0 con 0x7fae200f52d0 2023-11-16T14:05:01.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.701+0000 7fae15ffb700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 0x7fae0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:01.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.701+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fae1c0424a0 con 0x7fae200f52d0 2023-11-16T14:05:01.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.701+0000 7fae24e37700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 0x7fae0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:01.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.702+0000 7fae24e37700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 0x7fae0c033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fae100096a0 tx=0x7fae10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:01.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.706+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fae1c01d7f0 con 0x7fae200f52d0 2023-11-16T14:05:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.912+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae200f9ce0 con 0x7fae200f52d0 2023-11-16T14:05:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.913+0000 7fae15ffb700 1 -- 172.21.15.112:0/1182720522 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae1c01d7f0 con 0x7fae200f52d0 2023-11-16T14:05:01.915 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:01.915 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 msgr2=0x7fae0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 0x7fae0c033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fae100096a0 tx=0x7fae10006b40).stop 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 msgr2=0x7fae20108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fae1c0089f0 tx=0x7fae1c00b040).stop 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 shutdown_connections 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fae0c031090 0x7fae0c033550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 --2- 172.21.15.112:0/1182720522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae200f52d0 0x7fae20108800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 >> 172.21.15.112:0/1182720522 conn(0x7fae200f0a30 msgr2=0x7fae200f2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 shutdown_connections 2023-11-16T14:05:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:01.915+0000 7fae2663a700 1 -- 172.21.15.112:0/1182720522 wait complete. 2023-11-16T14:05:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:00.869001+0000 mgr.smithi061.zcjido (mgr.14156) 105 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:01.913363+0000 mon.smithi061 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.112:0/1182720522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:03.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:03.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:02.869376+0000 mgr.smithi061.zcjido (mgr.14156) 106 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:04.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 -- 172.21.15.112:0/1382560594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 msgr2=0x7f35140f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 --2- 172.21.15.112:0/1382560594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35140f56f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f3504009fb0 tx=0x7f3504009b30).stop 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 -- 172.21.15.112:0/1382560594 shutdown_connections 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 --2- 172.21.15.112:0/1382560594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35140f56f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 -- 172.21.15.112:0/1382560594 >> 172.21.15.112:0/1382560594 conn(0x7f35140f0a30 msgr2=0x7f35140f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 -- 172.21.15.112:0/1382560594 shutdown_connections 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.380+0000 7f351359e700 1 -- 172.21.15.112:0/1382560594 wait complete. 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351359e700 1 Processor -- start 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351359e700 1 -- start start 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:04.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514060360 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58378/0 (socket says 172.21.15.112:58378) 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351259c700 1 -- 172.21.15.112:0/1753872577 learned_addr learned my addr 172.21.15.112:0/1753872577 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.381+0000 7f351259c700 1 -- 172.21.15.112:0/1753872577 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504005040 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f351259c700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f35040089f0 tx=0x7f350400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3504009320 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3514060560 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3504013ec0 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3504014d30 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.382+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514109a00 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.383+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f35040263a0 con 0x7f35140f52d0 2023-11-16T14:05:04.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.383+0000 7f3502ffd700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 0x7f34fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:04.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.383+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f35040423d0 con 0x7f35140f52d0 2023-11-16T14:05:04.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.383+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35140591e0 con 0x7f35140f52d0 2023-11-16T14:05:04.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.383+0000 7f3511d9b700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 0x7f34fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:04.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.384+0000 7f3511d9b700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 0x7f34fc033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f35080096a0 tx=0x7f3508006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:04.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.388+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3504012920 con 0x7f35140f52d0 2023-11-16T14:05:04.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.594+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35140fadc0 con 0x7f35140f52d0 2023-11-16T14:05:04.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.594+0000 7f3502ffd700 1 -- 172.21.15.112:0/1753872577 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f350402b020 con 0x7f35140f52d0 2023-11-16T14:05:04.596 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:04.596 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:04.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 msgr2=0x7f34fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:04.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 0x7f34fc033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f35080096a0 tx=0x7f3508006b40).stop 2023-11-16T14:05:04.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 msgr2=0x7f35141092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:04.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f35040089f0 tx=0x7f350400b040).stop 2023-11-16T14:05:04.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 shutdown_connections 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f34fc031090 0x7f34fc033550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 --2- 172.21.15.112:0/1753872577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35140f52d0 0x7f35141092d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 >> 172.21.15.112:0/1753872577 conn(0x7f35140f0a30 msgr2=0x7f35140f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 shutdown_connections 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:04.596+0000 7f351359e700 1 -- 172.21.15.112:0/1753872577 wait complete. 2023-11-16T14:05:04.599 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:05.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:04 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:04.594805+0000 mon.smithi061 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.112:0/1753872577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:05.964 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:05.965 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:06.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:04.869697+0000 mgr.smithi061.zcjido (mgr.14156) 107 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:07.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.026+0000 7f841a3bb700 1 -- 172.21.15.112:0/472834707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 msgr2=0x7f84140f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:23.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.026+0000 7f841a3bb700 1 --2- 172.21.15.112:0/472834707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f84140f6800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8410009fb0 tx=0x7f8410009b30).stop 2023-11-16T14:05:23.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.026+0000 7f841a3bb700 1 -- 172.21.15.112:0/472834707 shutdown_connections 2023-11-16T14:05:23.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.026+0000 7f841a3bb700 1 --2- 172.21.15.112:0/472834707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f84140f6800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:23.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.026+0000 7f841a3bb700 1 -- 172.21.15.112:0/472834707 >> 172.21.15.112:0/472834707 conn(0x7f84140f1b40 msgr2=0x7f84140f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 -- 172.21.15.112:0/472834707 shutdown_connections 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 -- 172.21.15.112:0/472834707 wait complete. 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 Processor -- start 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 -- start start 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:23.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f841a3bb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8414069f00 con 0x7f84140f63e0 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f84193b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f84193b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42762/0 (socket says 172.21.15.112:42762) 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.027+0000 7f84193b9700 1 -- 172.21.15.112:0/4159989979 learned_addr learned my addr 172.21.15.112:0/4159989979 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f84193b9700 1 -- 172.21.15.112:0/4159989979 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410005040 con 0x7f84140f63e0 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f84193b9700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f84100089f0 tx=0x7f841000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:23.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8410009320 con 0x7f84140f63e0 2023-11-16T14:05:23.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841406a0a0 con 0x7f84140f63e0 2023-11-16T14:05:23.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8410013ec0 con 0x7f84140f63e0 2023-11-16T14:05:23.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8410014d30 con 0x7f84140f63e0 2023-11-16T14:05:23.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.028+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841406a530 con 0x7f84140f63e0 2023-11-16T14:05:23.872 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:06.869937+0000 mgr.smithi061.zcjido (mgr.14156) 108 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:07.242565+0000 mon.smithi061 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.112:0/4159989979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:08.870271+0000 mgr.smithi061.zcjido (mgr.14156) 109 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:10.870654+0000 mgr.smithi061.zcjido (mgr.14156) 110 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:12.870965+0000 mgr.smithi061.zcjido (mgr.14156) 111 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:14.871308+0000 mgr.smithi061.zcjido (mgr.14156) 112 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:16.871674+0000 mgr.smithi061.zcjido (mgr.14156) 113 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:18.872037+0000 mgr.smithi061.zcjido (mgr.14156) 114 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.907 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:20.872331+0000 mgr.smithi061.zcjido (mgr.14156) 115 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.029+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f84100095f0 con 0x7f84140f63e0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.029+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84140591e0 con 0x7f84140f63e0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.029+0000 7f8409ffb700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 0x7f8400033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.029+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f84100424a0 con 0x7f84140f63e0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.029+0000 7f8418bb8700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 0x7f8400033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.030+0000 7f8418bb8700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 0x7f8400033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f84040096a0 tx=0x7f8404006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.034+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f841001d7f0 con 0x7f84140f63e0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.241+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84140fbed0 con 0x7f84140f63e0 2023-11-16T14:05:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.242+0000 7f8409ffb700 1 -- 172.21.15.112:0/4159989979 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f841001d7f0 con 0x7f84140f63e0 2023-11-16T14:05:23.909 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:23.909 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 msgr2=0x7f8400033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 0x7f8400033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f84040096a0 tx=0x7f8404006b40).stop 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 msgr2=0x7f841406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f84100089f0 tx=0x7f841000b040).stop 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 shutdown_connections 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8400031090 0x7f8400033550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 --2- 172.21.15.112:0/4159989979 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84140f63e0 0x7f841406ba80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 >> 172.21.15.112:0/4159989979 conn(0x7f84140f1b40 msgr2=0x7f84140f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 shutdown_connections 2023-11-16T14:05:23.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:07.244+0000 7f841a3bb700 1 -- 172.21.15.112:0/4159989979 wait complete. 2023-11-16T14:05:23.961 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:22.872649+0000 mgr.smithi061.zcjido (mgr.14156) 116 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:24.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:24.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.022+0000 7fe7b1e87700 1 -- 172.21.15.112:0/2119510865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 msgr2=0x7fe7ac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.022+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/2119510865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac0f69b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe7a0004760 tx=0x7fe7a0009ad0).stop 2023-11-16T14:05:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.022+0000 7fe7b1e87700 1 -- 172.21.15.112:0/2119510865 shutdown_connections 2023-11-16T14:05:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.022+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/2119510865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac0f69b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.022+0000 7fe7b1e87700 1 -- 172.21.15.112:0/2119510865 >> 172.21.15.112:0/2119510865 conn(0x7fe7ac0f1cd0 msgr2=0x7fe7ac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 -- 172.21.15.112:0/2119510865 shutdown_connections 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 -- 172.21.15.112:0/2119510865 wait complete. 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 Processor -- start 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 -- start start 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.023+0000 7fe7b1e87700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac106cb0 con 0x7fe7ac0f6590 2023-11-16T14:05:26.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44202/0 (socket says 172.21.15.112:44202) 2023-11-16T14:05:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7ab7fe700 1 -- 172.21.15.112:0/3672452047 learned_addr learned my addr 172.21.15.112:0/3672452047 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7ab7fe700 1 -- 172.21.15.112:0/3672452047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a0009d70 con 0x7fe7ac0f6590 2023-11-16T14:05:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7ab7fe700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe7a0004170 tx=0x7fe7a0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a000e040 con 0x7fe7ac0f6590 2023-11-16T14:05:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac106eb0 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe7a0008ef0 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a0014c90 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.024+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac107340 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.025+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe7a0014420 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.025+0000 7fe793fff700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 0x7fe794033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.025+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac045bb0 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.025+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe7a0042260 con 0x7fe7ac0f6590 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.026+0000 7fe7aaffd700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 0x7fe794033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.026+0000 7fe7aaffd700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 0x7fe794033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe79c0096a0 tx=0x7fe79c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.030+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe7a0026300 con 0x7fe7ac0f6590 2023-11-16T14:05:26.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.239+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7ac1081c0 con 0x7fe7ac0f6590 2023-11-16T14:05:26.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.240+0000 7fe793fff700 1 -- 172.21.15.112:0/3672452047 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7a0012b20 con 0x7fe7ac0f6590 2023-11-16T14:05:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:26.243 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 msgr2=0x7fe794033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 0x7fe794033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe79c0096a0 tx=0x7fe79c006b40).stop 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 msgr2=0x7fe7ac1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe7a0004170 tx=0x7fe7a0009ad0).stop 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 shutdown_connections 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe794031090 0x7fe794033550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 --2- 172.21.15.112:0/3672452047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0f6590 0x7fe7ac1067c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 >> 172.21.15.112:0/3672452047 conn(0x7fe7ac0f1cd0 msgr2=0x7fe7ac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 shutdown_connections 2023-11-16T14:05:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:26.242+0000 7fe7b1e87700 1 -- 172.21.15.112:0/3672452047 wait complete. 2023-11-16T14:05:26.246 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:24.872948+0000 mgr.smithi061.zcjido (mgr.14156) 117 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:27.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:26.241446+0000 mon.smithi061 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.112:0/3672452047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:27.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:27.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:26.873291+0000 mgr.smithi061.zcjido (mgr.14156) 118 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:28.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.647+0000 7f317a15e700 1 -- 172.21.15.112:0/3693559896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 msgr2=0x7f31740f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.647+0000 7f317a15e700 1 --2- 172.21.15.112:0/3693559896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f31740f56f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3170009fb0 tx=0x7f3170009b30).stop 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.647+0000 7f317a15e700 1 -- 172.21.15.112:0/3693559896 shutdown_connections 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.647+0000 7f317a15e700 1 --2- 172.21.15.112:0/3693559896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f31740f56f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.647+0000 7f317a15e700 1 -- 172.21.15.112:0/3693559896 >> 172.21.15.112:0/3693559896 conn(0x7f31740f0a30 msgr2=0x7f31740f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 -- 172.21.15.112:0/3693559896 shutdown_connections 2023-11-16T14:05:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 -- 172.21.15.112:0/3693559896 wait complete. 2023-11-16T14:05:28.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 Processor -- start 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 -- start start 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.648+0000 7f317a15e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3174105a00 con 0x7f31740f52d0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317915c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317915c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44220/0 (socket says 172.21.15.112:44220) 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317915c700 1 -- 172.21.15.112:0/2140863931 learned_addr learned my addr 172.21.15.112:0/2140863931 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317915c700 1 -- 172.21.15.112:0/2140863931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170005040 con 0x7f31740f52d0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317915c700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f31700089f0 tx=0x7f317000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3170009320 con 0x7f31740f52d0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3170013ec0 con 0x7f31740f52d0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3174105c00 con 0x7f31740f52d0 2023-11-16T14:05:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.649+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3170014d30 con 0x7f31740f52d0 2023-11-16T14:05:28.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.650+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3174106090 con 0x7f31740f52d0 2023-11-16T14:05:28.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.650+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f31700095f0 con 0x7f31740f52d0 2023-11-16T14:05:28.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.650+0000 7f3169ffb700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 0x7f3160033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:28.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.650+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3174045bb0 con 0x7f31740f52d0 2023-11-16T14:05:28.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.651+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f31700424a0 con 0x7f31740f52d0 2023-11-16T14:05:28.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.651+0000 7f317895b700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 0x7f3160033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:28.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.651+0000 7f317895b700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 0x7f3160033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:28.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.655+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f317001d7f0 con 0x7f31740f52d0 2023-11-16T14:05:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.862+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3174106f10 con 0x7f31740f52d0 2023-11-16T14:05:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.862+0000 7f3169ffb700 1 -- 172.21.15.112:0/2140863931 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f317001d7f0 con 0x7f31740f52d0 2023-11-16T14:05:28.865 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:28.866 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 msgr2=0x7f3160033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 0x7f3160033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).stop 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 msgr2=0x7f3174105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f31700089f0 tx=0x7f317000b040).stop 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 shutdown_connections 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3160031090 0x7f3160033550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 --2- 172.21.15.112:0/2140863931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31740f52d0 0x7f3174105510 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 >> 172.21.15.112:0/2140863931 conn(0x7f31740f0a30 msgr2=0x7f31740f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 shutdown_connections 2023-11-16T14:05:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:28.864+0000 7f317a15e700 1 -- 172.21.15.112:0/2140863931 wait complete. 2023-11-16T14:05:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:29.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:28.863950+0000 mon.smithi061 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.112:0/2140863931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:30.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:30.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:28.873651+0000 mgr.smithi061.zcjido (mgr.14156) 119 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:31.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 -- 172.21.15.112:0/3306444102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 msgr2=0x7f391c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:31.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 --2- 172.21.15.112:0/3306444102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c0f69b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3910004760 tx=0x7f3910009ad0).stop 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 -- 172.21.15.112:0/3306444102 shutdown_connections 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 --2- 172.21.15.112:0/3306444102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c0f69b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 -- 172.21.15.112:0/3306444102 >> 172.21.15.112:0/3306444102 conn(0x7f391c0f1cd0 msgr2=0x7f391c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 -- 172.21.15.112:0/3306444102 shutdown_connections 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.301+0000 7f3921acd700 1 -- 172.21.15.112:0/3306444102 wait complete. 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.302+0000 7f3921acd700 1 Processor -- start 2023-11-16T14:05:31.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.302+0000 7f3921acd700 1 -- start start 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.302+0000 7f3921acd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.302+0000 7f3921acd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c106cb0 con 0x7f391c0f6590 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f391b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f391b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44240/0 (socket says 172.21.15.112:44240) 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f391b7fe700 1 -- 172.21.15.112:0/2076047038 learned_addr learned my addr 172.21.15.112:0/2076047038 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f391b7fe700 1 -- 172.21.15.112:0/2076047038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910009d70 con 0x7f391c0f6590 2023-11-16T14:05:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f391b7fe700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3910004170 tx=0x7f3910009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f391000e040 con 0x7f391c0f6590 2023-11-16T14:05:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c106eb0 con 0x7f391c0f6590 2023-11-16T14:05:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3910008ef0 con 0x7f391c0f6590 2023-11-16T14:05:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910014c90 con 0x7f391c0f6590 2023-11-16T14:05:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.303+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c107340 con 0x7f391c0f6590 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.304+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3910014420 con 0x7f391c0f6590 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.304+0000 7f3903fff700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 0x7f3904033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.304+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c045bb0 con 0x7f391c0f6590 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.304+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3910042220 con 0x7f391c0f6590 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.304+0000 7f391affd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 0x7f3904033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.305+0000 7f391affd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 0x7f3904033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f390c0096a0 tx=0x7f390c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:31.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.309+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3910012d00 con 0x7f391c0f6590 2023-11-16T14:05:31.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.518+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f391c0fc170 con 0x7f391c0f6590 2023-11-16T14:05:31.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.519+0000 7f3903fff700 1 -- 172.21.15.112:0/2076047038 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3910012b20 con 0x7f391c0f6590 2023-11-16T14:05:31.521 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:31.521 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 msgr2=0x7f3904033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 0x7f3904033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f390c0096a0 tx=0x7f390c006b40).stop 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 msgr2=0x7f391c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3910004170 tx=0x7f3910009ad0).stop 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 shutdown_connections 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3904031090 0x7f3904033550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 --2- 172.21.15.112:0/2076047038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f391c0f6590 0x7f391c1067c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 >> 172.21.15.112:0/2076047038 conn(0x7f391c0f1cd0 msgr2=0x7f391c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 shutdown_connections 2023-11-16T14:05:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:31.521+0000 7f3921acd700 1 -- 172.21.15.112:0/2076047038 wait complete. 2023-11-16T14:05:31.524 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:30.874088+0000 mgr.smithi061.zcjido (mgr.14156) 120 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:31.519241+0000 mon.smithi061 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.112:0/2076047038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:32.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:32.928 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 -- 172.21.15.112:0/2021931519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 msgr2=0x7f2bd00fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 --2- 172.21.15.112:0/2021931519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd00fb5e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2bc0009fb0 tx=0x7f2bc0009b30).stop 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 -- 172.21.15.112:0/2021931519 shutdown_connections 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 --2- 172.21.15.112:0/2021931519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd00fb5e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 -- 172.21.15.112:0/2021931519 >> 172.21.15.112:0/2021931519 conn(0x7f2bd00f0be0 msgr2=0x7f2bd00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 -- 172.21.15.112:0/2021931519 shutdown_connections 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.998+0000 7f2bd8522700 1 -- 172.21.15.112:0/2021931519 wait complete. 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd8522700 1 Processor -- start 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd8522700 1 -- start start 2023-11-16T14:05:34.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd8522700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd8522700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0108e60 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd62be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd62be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44248/0 (socket says 172.21.15.112:44248) 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:33.999+0000 7f2bd62be700 1 -- 172.21.15.112:0/626664597 learned_addr learned my addr 172.21.15.112:0/626664597 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bd62be700 1 -- 172.21.15.112:0/626664597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0005040 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bd62be700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f2bc0009f80 tx=0x7f2bc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc0009320 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2bc0013680 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd0109060 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc001eda0 con 0x7f2bd00602e0 2023-11-16T14:05:34.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.000+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd01094f0 con 0x7f2bd00602e0 2023-11-16T14:05:34.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2bc001e420 con 0x7f2bd00602e0 2023-11-16T14:05:34.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bc6ffd700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 0x7f2bbc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:34.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd0045bb0 con 0x7f2bd00602e0 2023-11-16T14:05:34.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2bc0042360 con 0x7f2bd00602e0 2023-11-16T14:05:34.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bd5abd700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 0x7f2bbc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:34.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.001+0000 7f2bd5abd700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 0x7f2bbc033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2bcc0096a0 tx=0x7f2bcc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:34.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.006+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2bc001d380 con 0x7f2bd00602e0 2023-11-16T14:05:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:32.874427+0000 mgr.smithi061.zcjido (mgr.14156) 121 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:34.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.217+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bd010a2f0 con 0x7f2bd00602e0 2023-11-16T14:05:34.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.218+0000 7f2bc6ffd700 1 -- 172.21.15.112:0/626664597 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bc001cb10 con 0x7f2bd00602e0 2023-11-16T14:05:34.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:34.220 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.219+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 msgr2=0x7f2bbc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.219+0000 7f2bd8522700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 0x7f2bbc033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2bcc0096a0 tx=0x7f2bcc006b40).stop 2023-11-16T14:05:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.219+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 msgr2=0x7f2bd0108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.219+0000 7f2bd8522700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f2bc0009f80 tx=0x7f2bc000b040).stop 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.219+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 shutdown_connections 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.220+0000 7f2bd8522700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2bbc031090 0x7f2bbc033550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.220+0000 7f2bd8522700 1 --2- 172.21.15.112:0/626664597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bd00602e0 0x7f2bd0108970 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.220+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 >> 172.21.15.112:0/626664597 conn(0x7f2bd00f0be0 msgr2=0x7f2bd00f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.220+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 shutdown_connections 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:34.220+0000 7f2bd8522700 1 -- 172.21.15.112:0/626664597 wait complete. 2023-11-16T14:05:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:35.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:34.218415+0000 mon.smithi061 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.112:0/626664597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:35.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:35.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:34.874790+0000 mgr.smithi061.zcjido (mgr.14156) 122 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:34.981194+0000 mon.smithi061 (mon.0) 188 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:05:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:34.981842+0000 mon.smithi061 (mon.0) 189 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:05:36.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 -- 172.21.15.112:0/1863094873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 msgr2=0x7fc4300f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:36.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 --2- 172.21.15.112:0/1863094873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4300f6760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc42c009fb0 tx=0x7fc42c009b30).stop 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 -- 172.21.15.112:0/1863094873 shutdown_connections 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 --2- 172.21.15.112:0/1863094873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4300f6760 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 -- 172.21.15.112:0/1863094873 >> 172.21.15.112:0/1863094873 conn(0x7fc4300f1aa0 msgr2=0x7fc4300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 -- 172.21.15.112:0/1863094873 shutdown_connections 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 -- 172.21.15.112:0/1863094873 wait complete. 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.669+0000 7fc436bf6700 1 Processor -- start 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc436bf6700 1 -- start start 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc436bf6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc436bf6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430106ac0 con 0x7fc4300f6340 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc435bf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc435bf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40464/0 (socket says 172.21.15.112:40464) 2023-11-16T14:05:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc435bf4700 1 -- 172.21.15.112:0/3293118295 learned_addr learned my addr 172.21.15.112:0/3293118295 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc435bf4700 1 -- 172.21.15.112:0/3293118295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c005040 con 0x7fc4300f6340 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.670+0000 7fc435bf4700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc42c0089f0 tx=0x7fc42c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c009320 con 0x7fc4300f6340 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430106cc0 con 0x7fc4300f6340 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc42c013ec0 con 0x7fc4300f6340 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c014d30 con 0x7fc4300f6340 2023-11-16T14:05:36.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc430107150 con 0x7fc4300f6340 2023-11-16T14:05:36.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.671+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc42c0095f0 con 0x7fc4300f6340 2023-11-16T14:05:36.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.672+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430045bb0 con 0x7fc4300f6340 2023-11-16T14:05:36.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.672+0000 7fc4267fc700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 0x7fc41c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:36.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.672+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc42c042450 con 0x7fc4300f6340 2023-11-16T14:05:36.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.672+0000 7fc4353f3700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 0x7fc41c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:36.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.672+0000 7fc4353f3700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 0x7fc41c0335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc4200096a0 tx=0x7fc420006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:36.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.677+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc42c0268c0 con 0x7fc4300f6340 2023-11-16T14:05:36.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.885+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc430108000 con 0x7fc4300f6340 2023-11-16T14:05:36.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.885+0000 7fc4267fc700 1 -- 172.21.15.112:0/3293118295 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc42c01d380 con 0x7fc4300f6340 2023-11-16T14:05:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:36.888 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:36.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 msgr2=0x7fc41c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:36.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 0x7fc41c0335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc4200096a0 tx=0x7fc420006b40).stop 2023-11-16T14:05:36.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 msgr2=0x7fc4301065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:36.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc42c0089f0 tx=0x7fc42c00b040).stop 2023-11-16T14:05:36.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 shutdown_connections 2023-11-16T14:05:36.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc41c0310e0 0x7fc41c0335a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:36.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 --2- 172.21.15.112:0/3293118295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4300f6340 0x7fc4301065d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:36.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 >> 172.21.15.112:0/3293118295 conn(0x7fc4300f1aa0 msgr2=0x7fc4300f2600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:36.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 shutdown_connections 2023-11-16T14:05:36.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:36.887+0000 7fc436bf6700 1 -- 172.21.15.112:0/3293118295 wait complete. 2023-11-16T14:05:36.891 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:37.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:36.885992+0000 mon.smithi061 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.112:0/3293118295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:38.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:38.242 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:36.875136+0000 mgr.smithi061.zcjido (mgr.14156) 123 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- 172.21.15.112:0/2345728632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 msgr2=0x7ff95c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 --2- 172.21.15.112:0/2345728632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c0f58a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ff94c009fb0 tx=0x7ff94c009b30).stop 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- 172.21.15.112:0/2345728632 shutdown_connections 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 --2- 172.21.15.112:0/2345728632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c0f58a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- 172.21.15.112:0/2345728632 >> 172.21.15.112:0/2345728632 conn(0x7ff95c0f0be0 msgr2=0x7ff95c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- 172.21.15.112:0/2345728632 shutdown_connections 2023-11-16T14:05:39.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- 172.21.15.112:0/2345728632 wait complete. 2023-11-16T14:05:39.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 Processor -- start 2023-11-16T14:05:39.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- start start 2023-11-16T14:05:39.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.356+0000 7ff962532700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c106ce0 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff95bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff95bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40476/0 (socket says 172.21.15.112:40476) 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff95bfff700 1 -- 172.21.15.112:0/2643214354 learned_addr learned my addr 172.21.15.112:0/2643214354 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff95bfff700 1 -- 172.21.15.112:0/2643214354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c005040 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff95bfff700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff94c0089f0 tx=0x7ff94c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff94c009320 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c106ee0 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff94c013ec0 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff94c014d30 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.357+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c107370 con 0x7ff95c0f5480 2023-11-16T14:05:39.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.358+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff94c014420 con 0x7ff95c0f5480 2023-11-16T14:05:39.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.358+0000 7ff958ff9700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 0x7ff9480335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:39.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.358+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff94c0424c0 con 0x7ff95c0f5480 2023-11-16T14:05:39.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.358+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c045bb0 con 0x7ff95c0f5480 2023-11-16T14:05:39.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.359+0000 7ff95b7fe700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 0x7ff9480335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:39.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.359+0000 7ff95b7fe700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 0x7ff9480335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff9440096a0 tx=0x7ff944006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:39.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.363+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff94c014750 con 0x7ff95c0f5480 2023-11-16T14:05:39.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.573+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff95c0fb060 con 0x7ff95c0f5480 2023-11-16T14:05:39.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.573+0000 7ff958ff9700 1 -- 172.21.15.112:0/2643214354 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff94c01d380 con 0x7ff95c0f5480 2023-11-16T14:05:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:39.575 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 msgr2=0x7ff9480335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 0x7ff9480335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff9440096a0 tx=0x7ff944006b40).stop 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 msgr2=0x7ff95c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff94c0089f0 tx=0x7ff94c00b040).stop 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 shutdown_connections 2023-11-16T14:05:39.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff9480310e0 0x7ff9480335a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 --2- 172.21.15.112:0/2643214354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff95c0f5480 0x7ff95c1067f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 >> 172.21.15.112:0/2643214354 conn(0x7ff95c0f0be0 msgr2=0x7ff95c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 shutdown_connections 2023-11-16T14:05:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:39.575+0000 7ff962532700 1 -- 172.21.15.112:0/2643214354 wait complete. 2023-11-16T14:05:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:38.875458+0000 mgr.smithi061.zcjido (mgr.14156) 124 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:39.573856+0000 mon.smithi061 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.112:0/2643214354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:40.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:40.936 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:41.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.983+0000 7f4c716ba700 1 -- 172.21.15.112:0/1713176689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 msgr2=0x7f4c6c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:41.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.983+0000 7f4c716ba700 1 --2- 172.21.15.112:0/1713176689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c0f69b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4c5c009fb0 tx=0x7f4c5c009b30).stop 2023-11-16T14:05:41.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- 172.21.15.112:0/1713176689 shutdown_connections 2023-11-16T14:05:41.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 --2- 172.21.15.112:0/1713176689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c0f69b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- 172.21.15.112:0/1713176689 >> 172.21.15.112:0/1713176689 conn(0x7f4c6c0f1cf0 msgr2=0x7f4c6c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- 172.21.15.112:0/1713176689 shutdown_connections 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- 172.21.15.112:0/1713176689 wait complete. 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 Processor -- start 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- start start 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.984+0000 7f4c716ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c106c70 con 0x7f4c6c0f6590 2023-11-16T14:05:41.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40498/0 (socket says 172.21.15.112:40498) 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c6affd700 1 -- 172.21.15.112:0/5480073 learned_addr learned my addr 172.21.15.112:0/5480073 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c6affd700 1 -- 172.21.15.112:0/5480073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c005040 con 0x7f4c6c0f6590 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c6affd700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f4c5c0089f0 tx=0x7f4c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c5c009320 con 0x7f4c6c0f6590 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c106e70 con 0x7f4c6c0f6590 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4c5c013ec0 con 0x7f4c6c0f6590 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c5c014d30 con 0x7f4c6c0f6590 2023-11-16T14:05:41.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.985+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c107300 con 0x7f4c6c0f6590 2023-11-16T14:05:41.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.986+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4c5c014420 con 0x7f4c6c0f6590 2023-11-16T14:05:41.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.986+0000 7f4c537fe700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 0x7f4c54033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:41.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.986+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4c5c0423e0 con 0x7f4c6c0f6590 2023-11-16T14:05:41.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.986+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c045bb0 con 0x7f4c6c0f6590 2023-11-16T14:05:41.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.986+0000 7f4c6a7fc700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 0x7f4c54033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:41.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.987+0000 7f4c6a7fc700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 0x7f4c54033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4c600096a0 tx=0x7f4c60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:41.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:41.991+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4c5c014750 con 0x7f4c6c0f6590 2023-11-16T14:05:42.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.201+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c6c108040 con 0x7f4c6c0f6590 2023-11-16T14:05:42.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.202+0000 7f4c537fe700 1 -- 172.21.15.112:0/5480073 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c5c01d7f0 con 0x7f4c6c0f6590 2023-11-16T14:05:42.204 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:42.205 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 msgr2=0x7f4c54033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 0x7f4c54033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4c600096a0 tx=0x7f4c60006b40).stop 2023-11-16T14:05:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 msgr2=0x7f4c6c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f4c5c0089f0 tx=0x7f4c5c00b040).stop 2023-11-16T14:05:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 shutdown_connections 2023-11-16T14:05:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c54031090 0x7f4c54033550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 --2- 172.21.15.112:0/5480073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c6c0f6590 0x7f4c6c106780 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 >> 172.21.15.112:0/5480073 conn(0x7f4c6c0f1cf0 msgr2=0x7f4c6c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 shutdown_connections 2023-11-16T14:05:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:42.204+0000 7f4c716ba700 1 -- 172.21.15.112:0/5480073 wait complete. 2023-11-16T14:05:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:40.875807+0000 mgr.smithi061.zcjido (mgr.14156) 125 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:43.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:42.202653+0000 mon.smithi061 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.112:0/5480073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:43.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:43.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:44.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:42.876160+0000 mgr.smithi061.zcjido (mgr.14156) 126 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:44.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 -- 172.21.15.112:0/3287489956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 msgr2=0x7f55680f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 --2- 172.21.15.112:0/3287489956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55680f58a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5564009fb0 tx=0x7f5564009b30).stop 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 -- 172.21.15.112:0/3287489956 shutdown_connections 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 --2- 172.21.15.112:0/3287489956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55680f58a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 -- 172.21.15.112:0/3287489956 >> 172.21.15.112:0/3287489956 conn(0x7f55680f0be0 msgr2=0x7f55680f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 -- 172.21.15.112:0/3287489956 shutdown_connections 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 -- 172.21.15.112:0/3287489956 wait complete. 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.642+0000 7f556f280700 1 Processor -- start 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556f280700 1 -- start start 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556f280700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556f280700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5568106cd0 con 0x7f55680f5480 2023-11-16T14:05:44.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556d01c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556d01c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40514/0 (socket says 172.21.15.112:40514) 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556d01c700 1 -- 172.21.15.112:0/2324083639 learned_addr learned my addr 172.21.15.112:0/2324083639 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556d01c700 1 -- 172.21.15.112:0/2324083639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5564005040 con 0x7f55680f5480 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556d01c700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5564000c00 tx=0x7f556400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55640095f0 con 0x7f55680f5480 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5568106ed0 con 0x7f55680f5480 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5564007e50 con 0x7f55680f5480 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.643+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5564014e70 con 0x7f55680f5480 2023-11-16T14:05:44.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.644+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5568107360 con 0x7f55680f5480 2023-11-16T14:05:44.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.644+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5564025070 con 0x7f55680f5480 2023-11-16T14:05:44.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.645+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5568045bb0 con 0x7f55680f5480 2023-11-16T14:05:44.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.645+0000 7f555dffb700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 0x7f5554033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:44.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.645+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5564041d50 con 0x7f55680f5480 2023-11-16T14:05:44.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.645+0000 7f556c81b700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 0x7f5554033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:44.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.645+0000 7f556c81b700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 0x7f5554033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f55580096a0 tx=0x7f5558006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:44.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.650+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5564014770 con 0x7f55680f5480 2023-11-16T14:05:44.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.858+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5568108160 con 0x7f55680f5480 2023-11-16T14:05:44.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.859+0000 7f555dffb700 1 -- 172.21.15.112:0/2324083639 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5564014770 con 0x7f55680f5480 2023-11-16T14:05:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:44.862 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:44.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 msgr2=0x7f5554033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 0x7f5554033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f55580096a0 tx=0x7f5558006b40).stop 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 msgr2=0x7f55681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5564000c00 tx=0x7f556400b040).stop 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 shutdown_connections 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5554031090 0x7f5554033550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 --2- 172.21.15.112:0/2324083639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55680f5480 0x7f55681067e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 >> 172.21.15.112:0/2324083639 conn(0x7f55680f0be0 msgr2=0x7f556805ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 shutdown_connections 2023-11-16T14:05:44.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:44.861+0000 7f556f280700 1 -- 172.21.15.112:0/2324083639 wait complete. 2023-11-16T14:05:44.865 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:45.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:44 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:44.860648+0000 mon.smithi061 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.112:0/2324083639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:46.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:46.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:46.259 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:44.876474+0000 mgr.smithi061.zcjido (mgr.14156) 127 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:47.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 -- 172.21.15.112:0/2053261704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 msgr2=0x7f93cc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 --2- 172.21.15.112:0/2053261704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc0f58a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f93b4009fb0 tx=0x7f93b4009b30).stop 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 -- 172.21.15.112:0/2053261704 shutdown_connections 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 --2- 172.21.15.112:0/2053261704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc0f58a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 -- 172.21.15.112:0/2053261704 >> 172.21.15.112:0/2053261704 conn(0x7f93cc0f0be0 msgr2=0x7f93cc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 -- 172.21.15.112:0/2053261704 shutdown_connections 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.295+0000 7f93d15da700 1 -- 172.21.15.112:0/2053261704 wait complete. 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93d15da700 1 Processor -- start 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93d15da700 1 -- start start 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93d15da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93d15da700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc107280 con 0x7f93cc0f5480 2023-11-16T14:05:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:35728/0 (socket says 172.21.15.112:35728) 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93caffd700 1 -- 172.21.15.112:0/1767740757 learned_addr learned my addr 172.21.15.112:0/1767740757 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.296+0000 7f93caffd700 1 -- 172.21.15.112:0/1767740757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4005040 con 0x7f93cc0f5480 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93caffd700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f93b40089f0 tx=0x7f93b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93b4009320 con 0x7f93cc0f5480 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc060300 con 0x7f93cc0f5480 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f93b4013ec0 con 0x7f93cc0f5480 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93b4014d30 con 0x7f93cc0f5480 2023-11-16T14:05:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.297+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc10bd80 con 0x7f93cc0f5480 2023-11-16T14:05:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f93b40095f0 con 0x7f93cc0f5480 2023-11-16T14:05:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93c37fe700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 0x7f93b8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f93b40424a0 con 0x7f93cc0f5480 2023-11-16T14:05:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc045bb0 con 0x7f93cc0f5480 2023-11-16T14:05:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93ca7fc700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 0x7f93b8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.298+0000 7f93ca7fc700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 0x7f93b8033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f93bc0096a0 tx=0x7f93bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.303+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f93b401d7f0 con 0x7f93cc0f5480 2023-11-16T14:05:47.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.511+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93cc10c430 con 0x7f93cc0f5480 2023-11-16T14:05:47.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.512+0000 7f93c37fe700 1 -- 172.21.15.112:0/1767740757 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93b401d7f0 con 0x7f93cc0f5480 2023-11-16T14:05:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:47.515 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 msgr2=0x7f93b8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 0x7f93b8033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f93bc0096a0 tx=0x7f93bc006b40).stop 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 msgr2=0x7f93cc10b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f93b40089f0 tx=0x7f93b400b040).stop 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 shutdown_connections 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f93b8031090 0x7f93b8033550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 --2- 172.21.15.112:0/1767740757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93cc0f5480 0x7f93cc10b890 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 >> 172.21.15.112:0/1767740757 conn(0x7f93cc0f0be0 msgr2=0x7f93cc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 shutdown_connections 2023-11-16T14:05:47.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:47.514+0000 7f93d15da700 1 -- 172.21.15.112:0/1767740757 wait complete. 2023-11-16T14:05:47.518 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:46.876812+0000 mgr.smithi061.zcjido (mgr.14156) 128 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:47.513373+0000 mon.smithi061 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.112:0/1767740757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:48.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:48.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:49.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 -- 172.21.15.112:0/1541971894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 msgr2=0x7fa7f40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:49.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/1541971894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f40f58a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa7e4009fb0 tx=0x7fa7e4009b30).stop 2023-11-16T14:05:49.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 -- 172.21.15.112:0/1541971894 shutdown_connections 2023-11-16T14:05:49.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/1541971894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f40f58a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:49.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 -- 172.21.15.112:0/1541971894 >> 172.21.15.112:0/1541971894 conn(0x7fa7f40f0be0 msgr2=0x7fa7f40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:49.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 -- 172.21.15.112:0/1541971894 shutdown_connections 2023-11-16T14:05:49.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.949+0000 7fa7f9e47700 1 -- 172.21.15.112:0/1541971894 wait complete. 2023-11-16T14:05:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f9e47700 1 Processor -- start 2023-11-16T14:05:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f9e47700 1 -- start start 2023-11-16T14:05:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f9e47700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f9e47700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f4108ef0 con 0x7fa7f40f5480 2023-11-16T14:05:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:35738/0 (socket says 172.21.15.112:35738) 2023-11-16T14:05:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f37fe700 1 -- 172.21.15.112:0/4159647508 learned_addr learned my addr 172.21.15.112:0/4159647508 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f37fe700 1 -- 172.21.15.112:0/4159647508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4005040 con 0x7fa7f40f5480 2023-11-16T14:05:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.950+0000 7fa7f37fe700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa7e40089f0 tx=0x7fa7e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:49.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e4009320 con 0x7fa7f40f5480 2023-11-16T14:05:49.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f41090f0 con 0x7fa7f40f5480 2023-11-16T14:05:49.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa7e4013ec0 con 0x7fa7f40f5480 2023-11-16T14:05:49.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e4014d30 con 0x7fa7f40f5480 2023-11-16T14:05:49.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f4109580 con 0x7fa7f40f5480 2023-11-16T14:05:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.951+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa7e40095f0 con 0x7fa7f40f5480 2023-11-16T14:05:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.952+0000 7fa7f0ff9700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 0x7fa7e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.952+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa7e40423e0 con 0x7fa7f40f5480 2023-11-16T14:05:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.952+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f4045bb0 con 0x7fa7f40f5480 2023-11-16T14:05:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.952+0000 7fa7ebfff700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 0x7fa7e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.952+0000 7fa7ebfff700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 0x7fa7e0033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa7dc0096a0 tx=0x7fa7dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:49.957+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa7e401dd70 con 0x7fa7f40f5480 2023-11-16T14:05:50.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.162+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7f40fb060 con 0x7fa7f40f5480 2023-11-16T14:05:50.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.163+0000 7fa7f0ff9700 1 -- 172.21.15.112:0/4159647508 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7e402b020 con 0x7fa7f40f5480 2023-11-16T14:05:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:50.166 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 msgr2=0x7fa7e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 0x7fa7e0033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa7dc0096a0 tx=0x7fa7dc006b40).stop 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 msgr2=0x7fa7f4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa7e40089f0 tx=0x7fa7e400b040).stop 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 shutdown_connections 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa7e0031090 0x7fa7e0033550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 --2- 172.21.15.112:0/4159647508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7f40f5480 0x7fa7f4108a00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 >> 172.21.15.112:0/4159647508 conn(0x7fa7f40f0be0 msgr2=0x7fa7f40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 shutdown_connections 2023-11-16T14:05:50.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:50.165+0000 7fa7f9e47700 1 -- 172.21.15.112:0/4159647508 wait complete. 2023-11-16T14:05:50.169 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:48.877197+0000 mgr.smithi061.zcjido (mgr.14156) 129 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:51.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:50.164465+0000 mon.smithi061 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.112:0/4159647508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:51.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:51.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:50.877514+0000 mgr.smithi061.zcjido (mgr.14156) 130 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 -- 172.21.15.112:0/3753185998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 msgr2=0x7f5c740f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 --2- 172.21.15.112:0/3753185998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c740f56f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5c64009fb0 tx=0x7f5c64009b30).stop 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 -- 172.21.15.112:0/3753185998 shutdown_connections 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 --2- 172.21.15.112:0/3753185998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c740f56f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 -- 172.21.15.112:0/3753185998 >> 172.21.15.112:0/3753185998 conn(0x7f5c740f0a30 msgr2=0x7f5c740f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 -- 172.21.15.112:0/3753185998 shutdown_connections 2023-11-16T14:05:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 -- 172.21.15.112:0/3753185998 wait complete. 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.610+0000 7f5c793ac700 1 Processor -- start 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c793ac700 1 -- start start 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c793ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c793ac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74060320 con 0x7f5c740f52d0 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c73fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c73fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:35756/0 (socket says 172.21.15.112:35756) 2023-11-16T14:05:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c73fff700 1 -- 172.21.15.112:0/401562650 learned_addr learned my addr 172.21.15.112:0/401562650 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.611+0000 7f5c73fff700 1 -- 172.21.15.112:0/401562650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c64005040 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c73fff700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f5c640089f0 tx=0x7f5c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c64009320 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c74060520 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c741099c0 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5c64013ec0 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.612+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c64014d00 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.613+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5c64014420 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.613+0000 7f5c70ff9700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 0x7f5c60033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.613+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5c640422d0 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.613+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58004fa0 con 0x7f5c740f52d0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.613+0000 7f5c737fe700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 0x7f5c60033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.614+0000 7f5c737fe700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 0x7f5c60033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5c5c0096a0 tx=0x7f5c5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.618+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5c640263a0 con 0x7f5c740f52d0 2023-11-16T14:05:52.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.825+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c58005670 con 0x7f5c740f52d0 2023-11-16T14:05:52.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.826+0000 7f5c70ff9700 1 -- 172.21.15.112:0/401562650 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c6402b020 con 0x7f5c740f52d0 2023-11-16T14:05:52.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:52.829 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:52.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 msgr2=0x7f5c60033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:52.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 0x7f5c60033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5c5c0096a0 tx=0x7f5c5c006b40).stop 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 msgr2=0x7f5c74109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f5c640089f0 tx=0x7f5c6400b040).stop 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 shutdown_connections 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5c60031090 0x7f5c60033550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 --2- 172.21.15.112:0/401562650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740f52d0 0x7f5c74109310 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 >> 172.21.15.112:0/401562650 conn(0x7f5c740f0a30 msgr2=0x7f5c740f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 shutdown_connections 2023-11-16T14:05:52.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:52.827+0000 7f5c793ac700 1 -- 172.21.15.112:0/401562650 wait complete. 2023-11-16T14:05:52.832 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:52.827241+0000 mon.smithi061 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.112:0/401562650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:52.877775+0000 mgr.smithi061.zcjido (mgr.14156) 131 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:54.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:54.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:55.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/570941256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 msgr2=0x7f3cdc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:55.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/570941256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc0f58a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3cc4009fb0 tx=0x7f3cc4009b30).stop 2023-11-16T14:05:55.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/570941256 shutdown_connections 2023-11-16T14:05:55.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/570941256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc0f58a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:55.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/570941256 >> 172.21.15.112:0/570941256 conn(0x7f3cdc0f0be0 msgr2=0x7f3cdc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:55.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/570941256 shutdown_connections 2023-11-16T14:05:55.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.285+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/570941256 wait complete. 2023-11-16T14:05:55.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3ce1ee0700 1 Processor -- start 2023-11-16T14:05:55.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3ce1ee0700 1 -- start start 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3ce1ee0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3ce1ee0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc105be0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3cdb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3cdb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60398/0 (socket says 172.21.15.112:60398) 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3cdb7fe700 1 -- 172.21.15.112:0/821752674 learned_addr learned my addr 172.21.15.112:0/821752674 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.286+0000 7f3cdb7fe700 1 -- 172.21.15.112:0/821752674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4005040 con 0x7f3cdc0f5480 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3cdb7fe700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3cc4000c00 tx=0x7f3cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc40095f0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc105de0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3cc4007e50 con 0x7f3cdc0f5480 2023-11-16T14:05:55.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc4014e70 con 0x7f3cdc0f5480 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.287+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc106270 con 0x7f3cdc0f5480 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3cc4025070 con 0x7f3cdc0f5480 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3cd3fff700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 0x7f3cc8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3cc4041d50 con 0x7f3cdc0f5480 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc045bb0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3cdaffd700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 0x7f3cc8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:55.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.288+0000 7f3cdaffd700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 0x7f3cc8033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3ccc0096a0 tx=0x7f3ccc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:55.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.293+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3cc40146b0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.501+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cdc0fb060 con 0x7f3cdc0f5480 2023-11-16T14:05:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.502+0000 7f3cd3fff700 1 -- 172.21.15.112:0/821752674 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cc40146b0 con 0x7f3cdc0f5480 2023-11-16T14:05:55.504 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:55.505 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 msgr2=0x7f3cc8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 0x7f3cc8033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3ccc0096a0 tx=0x7f3ccc006b40).stop 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 msgr2=0x7f3cdc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3cc4000c00 tx=0x7f3cc400b040).stop 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 shutdown_connections 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3cc8031090 0x7f3cc8033550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 --2- 172.21.15.112:0/821752674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cdc0f5480 0x7f3cdc1056f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.503+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 >> 172.21.15.112:0/821752674 conn(0x7f3cdc0f0be0 msgr2=0x7f3cdc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.504+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 shutdown_connections 2023-11-16T14:05:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:55.504+0000 7f3ce1ee0700 1 -- 172.21.15.112:0/821752674 wait complete. 2023-11-16T14:05:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:54.878138+0000 mgr.smithi061.zcjido (mgr.14156) 132 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:55.503129+0000 mon.smithi061 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.112:0/821752674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:56.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:56.863 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:05:57.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 -- 172.21.15.112:0/2937548561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 msgr2=0x7fb6dc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:57.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/2937548561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc0f58a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb6cc009fb0 tx=0x7fb6cc009b30).stop 2023-11-16T14:05:57.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 -- 172.21.15.112:0/2937548561 shutdown_connections 2023-11-16T14:05:57.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/2937548561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc0f58a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:57.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 -- 172.21.15.112:0/2937548561 >> 172.21.15.112:0/2937548561 conn(0x7fb6dc0f0be0 msgr2=0x7fb6dc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:57.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 -- 172.21.15.112:0/2937548561 shutdown_connections 2023-11-16T14:05:57.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.927+0000 7fb6e23a8700 1 -- 172.21.15.112:0/2937548561 wait complete. 2023-11-16T14:05:57.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6e23a8700 1 Processor -- start 2023-11-16T14:05:57.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6e23a8700 1 -- start start 2023-11-16T14:05:57.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6e23a8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:57.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6e23a8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc105be0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:57.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60404/0 (socket says 172.21.15.112:60404) 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.928+0000 7fb6dbfff700 1 -- 172.21.15.112:0/1832058331 learned_addr learned my addr 172.21.15.112:0/1832058331 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6dbfff700 1 -- 172.21.15.112:0/1832058331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc005040 con 0x7fb6dc0f5480 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6dbfff700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb6cc0089f0 tx=0x7fb6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc009320 con 0x7fb6dc0f5480 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc105de0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb6cc013ec0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc014d30 con 0x7fb6dc0f5480 2023-11-16T14:05:57.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.929+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc106270 con 0x7fb6dc0f5480 2023-11-16T14:05:57.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.930+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc045bb0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.931+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb6cc0095f0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.931+0000 7fb6d8ff9700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 0x7fb6c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:05:57.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.931+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb6cc0425e0 con 0x7fb6dc0f5480 2023-11-16T14:05:57.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.932+0000 7fb6db7fe700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 0x7fb6c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:05:57.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.932+0000 7fb6db7fe700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 0x7fb6c4033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:05:57.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:57.936+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb6cc00f990 con 0x7fb6dc0f5480 2023-11-16T14:05:58.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.144+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6dc0fb060 con 0x7fb6dc0f5480 2023-11-16T14:05:58.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.145+0000 7fb6d8ff9700 1 -- 172.21.15.112:0/1832058331 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6cc00f340 con 0x7fb6dc0f5480 2023-11-16T14:05:58.148 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:05:58.148 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:05:58.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 msgr2=0x7fb6c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:58.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 0x7fb6c4033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).stop 2023-11-16T14:05:58.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 msgr2=0x7fb6dc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:05:58.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb6cc0089f0 tx=0x7fb6cc00b040).stop 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 shutdown_connections 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb6c4031090 0x7fb6c4033550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 --2- 172.21.15.112:0/1832058331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6dc0f5480 0x7fb6dc1056f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 >> 172.21.15.112:0/1832058331 conn(0x7fb6dc0f0be0 msgr2=0x7fb6dc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 shutdown_connections 2023-11-16T14:05:58.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:05:58.146+0000 7fb6e23a8700 1 -- 172.21.15.112:0/1832058331 wait complete. 2023-11-16T14:05:58.153 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:05:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:56.878483+0000 mgr.smithi061.zcjido (mgr.14156) 133 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:05:59.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:58 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:05:58.146246+0000 mon.smithi061 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.112:0/1832058331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:05:59.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:05:59.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:05:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:05:58.878750+0000 mgr.smithi061.zcjido (mgr.14156) 134 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:00.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- 172.21.15.112:0/3260396309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 msgr2=0x7f81e80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:00.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 --2- 172.21.15.112:0/3260396309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e80f58a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f81d8009fb0 tx=0x7f81d8009b30).stop 2023-11-16T14:06:00.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- 172.21.15.112:0/3260396309 shutdown_connections 2023-11-16T14:06:00.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 --2- 172.21.15.112:0/3260396309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e80f58a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- 172.21.15.112:0/3260396309 >> 172.21.15.112:0/3260396309 conn(0x7f81e80f0be0 msgr2=0x7f81e80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- 172.21.15.112:0/3260396309 shutdown_connections 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- 172.21.15.112:0/3260396309 wait complete. 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 Processor -- start 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- start start 2023-11-16T14:06:00.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.564+0000 7f81ee36a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8107300 con 0x7f81e80f5480 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60410/0 (socket says 172.21.15.112:60410) 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e7fff700 1 -- 172.21.15.112:0/550195746 learned_addr learned my addr 172.21.15.112:0/550195746 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e7fff700 1 -- 172.21.15.112:0/550195746 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8005040 con 0x7f81e80f5480 2023-11-16T14:06:00.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e7fff700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f81d8000c00 tx=0x7f81d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:00.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d80095f0 con 0x7f81e80f5480 2023-11-16T14:06:00.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e8060300 con 0x7f81e80f5480 2023-11-16T14:06:00.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f81d8007e50 con 0x7f81e80f5480 2023-11-16T14:06:00.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d8014e70 con 0x7f81e80f5480 2023-11-16T14:06:00.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.565+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e810be00 con 0x7f81e80f5480 2023-11-16T14:06:00.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.566+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f81d80079c0 con 0x7f81e80f5480 2023-11-16T14:06:00.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.566+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e8045bb0 con 0x7f81e80f5480 2023-11-16T14:06:00.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.566+0000 7f81e4ff9700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 0x7f81d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:00.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.566+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f81d8041d10 con 0x7f81e80f5480 2023-11-16T14:06:00.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.567+0000 7f81e77fe700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 0x7f81d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:00.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.567+0000 7f81e77fe700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 0x7f81d0033550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f81dc0096a0 tx=0x7f81dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:00.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.571+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f81d801b910 con 0x7f81e80f5480 2023-11-16T14:06:00.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.781+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81e80faf70 con 0x7f81e80f5480 2023-11-16T14:06:00.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.782+0000 7f81e4ff9700 1 -- 172.21.15.112:0/550195746 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81d801b2c0 con 0x7f81e80f5480 2023-11-16T14:06:00.785 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:00.785 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 msgr2=0x7f81d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 0x7f81d0033550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f81dc0096a0 tx=0x7f81dc006b40).stop 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 msgr2=0x7f81e810b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f81d8000c00 tx=0x7f81d800b040).stop 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 shutdown_connections 2023-11-16T14:06:00.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f81d0031090 0x7f81d0033550 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:00.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 --2- 172.21.15.112:0/550195746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81e80f5480 0x7f81e810b910 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:00.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 >> 172.21.15.112:0/550195746 conn(0x7f81e80f0be0 msgr2=0x7f81e80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:00.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 shutdown_connections 2023-11-16T14:06:00.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:00.784+0000 7f81ee36a700 1 -- 172.21.15.112:0/550195746 wait complete. 2023-11-16T14:06:00.788 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:01.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:00 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:00.783476+0000 mon.smithi061 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.112:0/550195746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:02.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:02.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:00.879024+0000 mgr.smithi061.zcjido (mgr.14156) 135 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 -- 172.21.15.112:0/1881966045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 msgr2=0x7fd85c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 --2- 172.21.15.112:0/1881966045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c0f9160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd84c009fb0 tx=0x7fd84c009b30).stop 2023-11-16T14:06:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 -- 172.21.15.112:0/1881966045 shutdown_connections 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 --2- 172.21.15.112:0/1881966045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c0f9160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 -- 172.21.15.112:0/1881966045 >> 172.21.15.112:0/1881966045 conn(0x7fd85c0f0be0 msgr2=0x7fd85c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 -- 172.21.15.112:0/1881966045 shutdown_connections 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 -- 172.21.15.112:0/1881966045 wait complete. 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.221+0000 7fd861f76700 1 Processor -- start 2023-11-16T14:06:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd861f76700 1 -- start start 2023-11-16T14:06:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd861f76700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd861f76700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c106c40 con 0x7fd85c060360 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60442/0 (socket says 172.21.15.112:60442) 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd85b7fe700 1 -- 172.21.15.112:0/1517514132 learned_addr learned my addr 172.21.15.112:0/1517514132 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.222+0000 7fd85b7fe700 1 -- 172.21.15.112:0/1517514132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c005040 con 0x7fd85c060360 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd85b7fe700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd84c000c00 tx=0x7fd84c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84c0095f0 con 0x7fd85c060360 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c106e40 con 0x7fd85c060360 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd84c007e50 con 0x7fd85c060360 2023-11-16T14:06:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84c014e70 con 0x7fd85c060360 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.223+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c1072d0 con 0x7fd85c060360 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.224+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd84c025070 con 0x7fd85c060360 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.224+0000 7fd843fff700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 0x7fd844033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.224+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd84c041e30 con 0x7fd85c060360 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.224+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c0591e0 con 0x7fd85c060360 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.224+0000 7fd85affd700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 0x7fd844033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:03.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.225+0000 7fd85affd700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 0x7fd844033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd8500096a0 tx=0x7fd850006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:03.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.229+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd84c026350 con 0x7fd85c060360 2023-11-16T14:06:03.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.438+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd85c0fc060 con 0x7fd85c060360 2023-11-16T14:06:03.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.439+0000 7fd843fff700 1 -- 172.21.15.112:0/1517514132 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd84c02d090 con 0x7fd85c060360 2023-11-16T14:06:03.442 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:03.442 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 msgr2=0x7fd844033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 0x7fd844033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd8500096a0 tx=0x7fd850006b40).stop 2023-11-16T14:06:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 msgr2=0x7fd85c106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd84c000c00 tx=0x7fd84c00b040).stop 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 shutdown_connections 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd844031090 0x7fd844033550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 --2- 172.21.15.112:0/1517514132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85c060360 0x7fd85c106750 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.440+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 >> 172.21.15.112:0/1517514132 conn(0x7fd85c0f0be0 msgr2=0x7fd85c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.441+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 shutdown_connections 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:03.441+0000 7fd861f76700 1 -- 172.21.15.112:0/1517514132 wait complete. 2023-11-16T14:06:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:02.879305+0000 mgr.smithi061.zcjido (mgr.14156) 136 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:03.440184+0000 mon.smithi061 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.112:0/1517514132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:04.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:04.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:05.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 -- 172.21.15.112:0/1966280340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 msgr2=0x7f3c300f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:05.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1966280340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c300f58a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3c2c009fb0 tx=0x7f3c2c009b30).stop 2023-11-16T14:06:05.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 -- 172.21.15.112:0/1966280340 shutdown_connections 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1966280340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c300f58a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 -- 172.21.15.112:0/1966280340 >> 172.21.15.112:0/1966280340 conn(0x7f3c300f0be0 msgr2=0x7f3c300f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 -- 172.21.15.112:0/1966280340 shutdown_connections 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.901+0000 7f3c373bd700 1 -- 172.21.15.112:0/1966280340 wait complete. 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c373bd700 1 Processor -- start 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c373bd700 1 -- start start 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c373bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c373bd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30060320 con 0x7f3c300f5480 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c35159700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c35159700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36230/0 (socket says 172.21.15.112:36230) 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c35159700 1 -- 172.21.15.112:0/1068114665 learned_addr learned my addr 172.21.15.112:0/1068114665 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:05.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.902+0000 7f3c35159700 1 -- 172.21.15.112:0/1068114665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c005040 con 0x7f3c300f5480 2023-11-16T14:06:05.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c35159700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f3c2c0089f0 tx=0x7f3c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:05.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c2c009320 con 0x7f3c300f5480 2023-11-16T14:06:05.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c30060520 con 0x7f3c300f5480 2023-11-16T14:06:05.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3c2c013ec0 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c2c014d30 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.903+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c30109ba0 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.904+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3c2c0095f0 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.904+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c30045bb0 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.904+0000 7f3c25ffb700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 0x7f3c1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.904+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3c2c042420 con 0x7f3c300f5480 2023-11-16T14:06:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.905+0000 7f3c34958700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 0x7f3c1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:05.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.905+0000 7f3c34958700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 0x7f3c1c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3c200096a0 tx=0x7f3c20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:05.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:05.910+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3c2c01d7f0 con 0x7f3c300f5480 2023-11-16T14:06:06.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.117+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c300faf70 con 0x7f3c300f5480 2023-11-16T14:06:06.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.118+0000 7f3c25ffb700 1 -- 172.21.15.112:0/1068114665 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c2c00fe40 con 0x7f3c300f5480 2023-11-16T14:06:06.121 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:06.121 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:06.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 msgr2=0x7f3c1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:06.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 0x7f3c1c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3c200096a0 tx=0x7f3c20006b40).stop 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 msgr2=0x7f3c301094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f3c2c0089f0 tx=0x7f3c2c00b040).stop 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 shutdown_connections 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3c1c031090 0x7f3c1c033550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 --2- 172.21.15.112:0/1068114665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c300f5480 0x7f3c301094f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:06.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 >> 172.21.15.112:0/1068114665 conn(0x7f3c300f0be0 msgr2=0x7f3c300f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:06.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 shutdown_connections 2023-11-16T14:06:06.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:06.119+0000 7f3c373bd700 1 -- 172.21.15.112:0/1068114665 wait complete. 2023-11-16T14:06:06.124 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:04.879649+0000 mgr.smithi061.zcjido (mgr.14156) 137 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:07.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:06 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:06.119224+0000 mon.smithi061 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.112:0/1068114665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:07.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:07.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:06.879947+0000 mgr.smithi061.zcjido (mgr.14156) 138 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:08.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.555+0000 7f1bfbfff700 1 -- 172.21.15.112:0/1360330897 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 msgr2=0x7f1bfc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:08.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.555+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/1360330897 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc0f56f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f1bec009fb0 tx=0x7f1bec009b30).stop 2023-11-16T14:06:08.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.555+0000 7f1bfbfff700 1 -- 172.21.15.112:0/1360330897 shutdown_connections 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.555+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/1360330897 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc0f56f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.555+0000 7f1bfbfff700 1 -- 172.21.15.112:0/1360330897 >> 172.21.15.112:0/1360330897 conn(0x7f1bfc0f0a30 msgr2=0x7f1bfc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 -- 172.21.15.112:0/1360330897 shutdown_connections 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 -- 172.21.15.112:0/1360330897 wait complete. 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 Processor -- start 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 -- start start 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.556+0000 7f1bfbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc106af0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:08.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36236/0 (socket says 172.21.15.112:36236) 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfaffd700 1 -- 172.21.15.112:0/2070041607 learned_addr learned my addr 172.21.15.112:0/2070041607 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfaffd700 1 -- 172.21.15.112:0/2070041607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec005040 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfaffd700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f1bec0089f0 tx=0x7f1bec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bec009320 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc106cf0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1bec013ec0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bec014d30 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.557+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc107180 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.558+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1bec0095f0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.558+0000 7f1be37fe700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 0x7f1be4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:08.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.558+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc045bb0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.558+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1bec042420 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.559+0000 7f1bfa7fc700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 0x7f1be4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:08.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.559+0000 7f1bfa7fc700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 0x7f1be4033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1bf00096a0 tx=0x7f1bf0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:08.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.564+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1bec00fe40 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.775+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bfc107fd0 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.775+0000 7f1be37fe700 1 -- 172.21.15.112:0/2070041607 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bec02c090 con 0x7f1bfc0f52d0 2023-11-16T14:06:08.778 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:08.778 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:08.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 msgr2=0x7f1be4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:08.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 0x7f1be4033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1bf00096a0 tx=0x7f1bf0006b40).stop 2023-11-16T14:06:08.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 msgr2=0x7f1bfc106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:08.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f1bec0089f0 tx=0x7f1bec00b040).stop 2023-11-16T14:06:08.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 shutdown_connections 2023-11-16T14:06:08.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1be4031090 0x7f1be4033550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:08.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 --2- 172.21.15.112:0/2070041607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bfc0f52d0 0x7f1bfc106600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:08.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 >> 172.21.15.112:0/2070041607 conn(0x7f1bfc0f0a30 msgr2=0x7f1bfc05fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:08.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 shutdown_connections 2023-11-16T14:06:08.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:08.777+0000 7f1bfbfff700 1 -- 172.21.15.112:0/2070041607 wait complete. 2023-11-16T14:06:08.781 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:09.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:08.775949+0000 mon.smithi061 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.112:0/2070041607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:10.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:10.107 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:08.880268+0000 mgr.smithi061.zcjido (mgr.14156) 139 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:11.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 -- 172.21.15.112:0/3174052027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 msgr2=0x7f14f40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 --2- 172.21.15.112:0/3174052027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f40f69b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f14e8009fb0 tx=0x7f14e8009b30).stop 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 -- 172.21.15.112:0/3174052027 shutdown_connections 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 --2- 172.21.15.112:0/3174052027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f40f69b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 -- 172.21.15.112:0/3174052027 >> 172.21.15.112:0/3174052027 conn(0x7f14f40f1cf0 msgr2=0x7f14f40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 -- 172.21.15.112:0/3174052027 shutdown_connections 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.193+0000 7f14f98fb700 1 -- 172.21.15.112:0/3174052027 wait complete. 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f98fb700 1 Processor -- start 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f98fb700 1 -- start start 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f98fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f98fb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4069f00 con 0x7f14f40f6590 2023-11-16T14:06:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36252/0 (socket says 172.21.15.112:36252) 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f2ffd700 1 -- 172.21.15.112:0/983267135 learned_addr learned my addr 172.21.15.112:0/983267135 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.194+0000 7f14f2ffd700 1 -- 172.21.15.112:0/983267135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8005040 con 0x7f14f40f6590 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14f2ffd700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f14e80089f0 tx=0x7f14e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e8009320 con 0x7f14f40f6590 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f14e8013ec0 con 0x7f14f40f6590 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f406a0a0 con 0x7f14f40f6590 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e8014d30 con 0x7f14f40f6590 2023-11-16T14:06:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.195+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f406a530 con 0x7f14f40f6590 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f14e80095f0 con 0x7f14f40f6590 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f40591e0 con 0x7f14f40f6590 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14db7fe700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 0x7f14dc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f14e80423a0 con 0x7f14f40f6590 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14f27fc700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 0x7f14dc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.196+0000 7f14f27fc700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 0x7f14dc0335a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.201+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f14e801d7f0 con 0x7f14f40f6590 2023-11-16T14:06:11.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.408+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14f40fc080 con 0x7f14f40f6590 2023-11-16T14:06:11.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.409+0000 7f14db7fe700 1 -- 172.21.15.112:0/983267135 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14e800fe40 con 0x7f14f40f6590 2023-11-16T14:06:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:11.411 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:11.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 msgr2=0x7f14dc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:11.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 0x7f14dc0335a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).stop 2023-11-16T14:06:11.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 msgr2=0x7f14f406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f14e80089f0 tx=0x7f14e800b040).stop 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 shutdown_connections 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f14dc0310e0 0x7f14dc0335a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 --2- 172.21.15.112:0/983267135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14f40f6590 0x7f14f406ba80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 >> 172.21.15.112:0/983267135 conn(0x7f14f40f1cf0 msgr2=0x7f14f40f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 shutdown_connections 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:11.410+0000 7f14f98fb700 1 -- 172.21.15.112:0/983267135 wait complete. 2023-11-16T14:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:10.880561+0000 mgr.smithi061.zcjido (mgr.14156) 140 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:11.409346+0000 mon.smithi061 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.112:0/983267135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:12.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:12.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:13.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 -- 172.21.15.112:0/521303532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 msgr2=0x7fbda00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 --2- 172.21.15.112:0/521303532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda00f58d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fbd90009fb0 tx=0x7fbd90009b30).stop 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 -- 172.21.15.112:0/521303532 shutdown_connections 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 --2- 172.21.15.112:0/521303532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda00f58d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 -- 172.21.15.112:0/521303532 >> 172.21.15.112:0/521303532 conn(0x7fbda00f0c10 msgr2=0x7fbda00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 -- 172.21.15.112:0/521303532 shutdown_connections 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 -- 172.21.15.112:0/521303532 wait complete. 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.818+0000 7fbda773b700 1 Processor -- start 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda773b700 1 -- start start 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda773b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:13.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda773b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda0060300 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda54d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda54d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36272/0 (socket says 172.21.15.112:36272) 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda54d7700 1 -- 172.21.15.112:0/1556145863 learned_addr learned my addr 172.21.15.112:0/1556145863 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda54d7700 1 -- 172.21.15.112:0/1556145863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd90005040 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda54d7700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fbd90000c00 tx=0x7fbd9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd900095f0 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda0060500 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd90007e50 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.819+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd90014e70 con 0x7fbda00f54b0 2023-11-16T14:06:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.820+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda0109bd0 con 0x7fbda00f54b0 2023-11-16T14:06:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbd90025070 con 0x7fbda00f54b0 2023-11-16T14:06:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00591e0 con 0x7fbda00f54b0 2023-11-16T14:06:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbd95ffb700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 0x7fbd8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd90041d10 con 0x7fbda00f54b0 2023-11-16T14:06:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbda4cd6700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 0x7fbd8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.821+0000 7fbda4cd6700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 0x7fbd8c033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbd9c0096a0 tx=0x7fbd9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:13.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:13.826+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd9000f4e0 con 0x7fbda00f54b0 2023-11-16T14:06:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:12.880865+0000 mgr.smithi061.zcjido (mgr.14156) 141 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:14.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.032+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbda0045bb0 con 0x7fbda00f54b0 2023-11-16T14:06:14.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.033+0000 7fbd95ffb700 1 -- 172.21.15.112:0/1556145863 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd9000f300 con 0x7fbda00f54b0 2023-11-16T14:06:14.035 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:14.035 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 msgr2=0x7fbd8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 0x7fbd8c033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbd9c0096a0 tx=0x7fbd9c006b40).stop 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 msgr2=0x7fbda0109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fbd90000c00 tx=0x7fbd9000b040).stop 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 shutdown_connections 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fbd8c031090 0x7fbd8c033550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 --2- 172.21.15.112:0/1556145863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbda00f54b0 0x7fbda0109520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 >> 172.21.15.112:0/1556145863 conn(0x7fbda00f0c10 msgr2=0x7fbda00f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 shutdown_connections 2023-11-16T14:06:14.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:14.035+0000 7fbda773b700 1 -- 172.21.15.112:0/1556145863 wait complete. 2023-11-16T14:06:14.038 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:15.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:14.033419+0000 mon.smithi061 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.112:0/1556145863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:15.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:15.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:16.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:14.881229+0000 mgr.smithi061.zcjido (mgr.14156) 142 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:16.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- 172.21.15.112:0/436136930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 msgr2=0x7f40540f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:16.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 --2- 172.21.15.112:0/436136930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40540f69b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4044004760 tx=0x7f4044009ad0).stop 2023-11-16T14:06:16.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- 172.21.15.112:0/436136930 shutdown_connections 2023-11-16T14:06:16.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 --2- 172.21.15.112:0/436136930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40540f69b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- 172.21.15.112:0/436136930 >> 172.21.15.112:0/436136930 conn(0x7f40540f1cd0 msgr2=0x7f40540f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- 172.21.15.112:0/436136930 shutdown_connections 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- 172.21.15.112:0/436136930 wait complete. 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 Processor -- start 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- start start 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405b2d2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4054106cb0 con 0x7f40540f6590 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405906e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405906e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38460/0 (socket says 172.21.15.112:38460) 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.531+0000 7f405906e700 1 -- 172.21.15.112:0/1531917681 learned_addr learned my addr 172.21.15.112:0/1531917681 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f405906e700 1 -- 172.21.15.112:0/1531917681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044009d70 con 0x7f40540f6590 2023-11-16T14:06:16.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f405906e700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f4044004170 tx=0x7f4044009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:16.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404400e040 con 0x7f40540f6590 2023-11-16T14:06:16.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4044008ef0 con 0x7f40540f6590 2023-11-16T14:06:16.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4054106eb0 con 0x7f40540f6590 2023-11-16T14:06:16.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4044014c90 con 0x7f40540f6590 2023-11-16T14:06:16.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.532+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4054107340 con 0x7f40540f6590 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.533+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4044014420 con 0x7f40540f6590 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.533+0000 7f4049ffb700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 0x7f4040033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.533+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4054045bb0 con 0x7f40540f6590 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.533+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4044042300 con 0x7f40540f6590 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.533+0000 7f405886d700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 0x7f4040033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:16.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.534+0000 7f405886d700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 0x7f4040033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f40500096a0 tx=0x7f4050006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:16.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.538+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4044012b20 con 0x7f40540f6590 2023-11-16T14:06:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.747+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40541081c0 con 0x7f40540f6590 2023-11-16T14:06:16.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.748+0000 7f4049ffb700 1 -- 172.21.15.112:0/1531917681 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4044012b20 con 0x7f40540f6590 2023-11-16T14:06:16.750 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:16.750 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:16.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 msgr2=0x7f4040033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 0x7f4040033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f40500096a0 tx=0x7f4050006b40).stop 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 msgr2=0x7f40541067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f4044004170 tx=0x7f4044009ad0).stop 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 shutdown_connections 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4040031090 0x7f4040033550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 --2- 172.21.15.112:0/1531917681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40540f6590 0x7f40541067c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 >> 172.21.15.112:0/1531917681 conn(0x7f40540f1cd0 msgr2=0x7f40540f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 shutdown_connections 2023-11-16T14:06:16.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:16.750+0000 7f405b2d2700 1 -- 172.21.15.112:0/1531917681 wait complete. 2023-11-16T14:06:16.753 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:17.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:16 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:16.748757+0000 mon.smithi061 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.112:0/1531917681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:18.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:18.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:18.257 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:16.881563+0000 mgr.smithi061.zcjido (mgr.14156) 143 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 -- 172.21.15.112:0/3976800544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 msgr2=0x7efe280fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 --2- 172.21.15.112:0/3976800544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe280fb610 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7efe1c009fb0 tx=0x7efe1c009b30).stop 2023-11-16T14:06:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 -- 172.21.15.112:0/3976800544 shutdown_connections 2023-11-16T14:06:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 --2- 172.21.15.112:0/3976800544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe280fb610 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 -- 172.21.15.112:0/3976800544 >> 172.21.15.112:0/3976800544 conn(0x7efe280f0c10 msgr2=0x7efe280f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 -- 172.21.15.112:0/3976800544 shutdown_connections 2023-11-16T14:06:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.170+0000 7efe2e094700 1 -- 172.21.15.112:0/3976800544 wait complete. 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe2e094700 1 Processor -- start 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe2e094700 1 -- start start 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe2e094700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe2e094700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28108e90 con 0x7efe280602e0 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38486/0 (socket says 172.21.15.112:38486) 2023-11-16T14:06:19.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.171+0000 7efe277fe700 1 -- 172.21.15.112:0/2716176530 learned_addr learned my addr 172.21.15.112:0/2716176530 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe277fe700 1 -- 172.21.15.112:0/2716176530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c005040 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe277fe700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7efe1c009f80 tx=0x7efe1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe1c009320 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe28109090 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efe1c013680 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe1c01eda0 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.172+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe28109520 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.173+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efe1c014420 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.173+0000 7efe0ffff700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 0x7efe10033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.173+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efe1c042380 con 0x7efe280602e0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.173+0000 7efe26ffd700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 0x7efe10033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.173+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe280591e0 con 0x7efe280602e0 2023-11-16T14:06:19.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.174+0000 7efe26ffd700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 0x7efe10033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.178+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efe1c01c910 con 0x7efe280602e0 2023-11-16T14:06:19.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.387+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe280fe290 con 0x7efe280602e0 2023-11-16T14:06:19.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.388+0000 7efe0ffff700 1 -- 172.21.15.112:0/2716176530 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe1c01c320 con 0x7efe280602e0 2023-11-16T14:06:19.390 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:19.390 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 msgr2=0x7efe10033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 0x7efe10033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).stop 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 msgr2=0x7efe281089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7efe1c009f80 tx=0x7efe1c00b040).stop 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 shutdown_connections 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efe10031090 0x7efe10033550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 --2- 172.21.15.112:0/2716176530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe280602e0 0x7efe281089a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:19.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 >> 172.21.15.112:0/2716176530 conn(0x7efe280f0c10 msgr2=0x7efe280f16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:19.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 shutdown_connections 2023-11-16T14:06:19.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:19.390+0000 7efe2e094700 1 -- 172.21.15.112:0/2716176530 wait complete. 2023-11-16T14:06:19.393 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:18.881829+0000 mgr.smithi061.zcjido (mgr.14156) 144 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:19.388613+0000 mon.smithi061 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.112:0/2716176530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:20.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:22.002 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:20.882184+0000 mgr.smithi061.zcjido (mgr.14156) 145 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.160+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/1737181955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 msgr2=0x7f3b740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.160+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/1737181955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b740f5830 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f3b64009fb0 tx=0x7f3b64009b30).stop 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.160+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/1737181955 shutdown_connections 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.160+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/1737181955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b740f5830 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.160+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/1737181955 >> 172.21.15.112:0/1737181955 conn(0x7f3b740f0b70 msgr2=0x7f3b740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/1737181955 shutdown_connections 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/1737181955 wait complete. 2023-11-16T14:06:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 Processor -- start 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 -- start start 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7b2e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b74106fb0 con 0x7f3b740f5410 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7907d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7907d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:38502/0 (socket says 172.21.15.112:38502) 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.161+0000 7f3b7907d700 1 -- 172.21.15.112:0/2664379110 learned_addr learned my addr 172.21.15.112:0/2664379110 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b7907d700 1 -- 172.21.15.112:0/2664379110 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b64005040 con 0x7f3b740f5410 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b7907d700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f3b640089f0 tx=0x7f3b6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b64009320 con 0x7f3b740f5410 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b741071b0 con 0x7f3b740f5410 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3b64013ec0 con 0x7f3b740f5410 2023-11-16T14:06:23.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b64014d30 con 0x7f3b740f5410 2023-11-16T14:06:23.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.162+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b74107640 con 0x7f3b740f5410 2023-11-16T14:06:23.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.163+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3b64014420 con 0x7f3b740f5410 2023-11-16T14:06:23.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.163+0000 7f3b69ffb700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 0x7f3b600335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:23.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.163+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b74045bb0 con 0x7f3b740f5410 2023-11-16T14:06:23.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.163+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3b640424c0 con 0x7f3b740f5410 2023-11-16T14:06:23.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.163+0000 7f3b7887c700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 0x7f3b600335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:23.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.164+0000 7f3b7887c700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 0x7f3b600335a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3b700096a0 tx=0x7f3b70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:23.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.168+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3b6401d980 con 0x7f3b740f5410 2023-11-16T14:06:23.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.376+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b7405eb30 con 0x7f3b740f5410 2023-11-16T14:06:23.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.377+0000 7f3b69ffb700 1 -- 172.21.15.112:0/2664379110 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b6401d380 con 0x7f3b740f5410 2023-11-16T14:06:23.380 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:23.380 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:23.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 msgr2=0x7f3b600335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:23.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 0x7f3b600335a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3b700096a0 tx=0x7f3b70006b40).stop 2023-11-16T14:06:23.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 msgr2=0x7f3b74106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f3b640089f0 tx=0x7f3b6400b040).stop 2023-11-16T14:06:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 shutdown_connections 2023-11-16T14:06:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b600310e0 0x7f3b600335a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 --2- 172.21.15.112:0/2664379110 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b740f5410 0x7f3b74106ac0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 >> 172.21.15.112:0/2664379110 conn(0x7f3b740f0b70 msgr2=0x7f3b7405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:23.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 shutdown_connections 2023-11-16T14:06:23.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:23.378+0000 7f3b7b2e1700 1 -- 172.21.15.112:0/2664379110 wait complete. 2023-11-16T14:06:23.383 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:22.882524+0000 mgr.smithi061.zcjido (mgr.14156) 146 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:23.378140+0000 mon.smithi061 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.112:0/2664379110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:24.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:24.741 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:25.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.985+0000 7f6d81199700 1 -- 172.21.15.112:0/1249763577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c060360 msgr2=0x7f6d7c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:25.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.985+0000 7f6d81199700 1 --2- 172.21.15.112:0/1249763577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c060360 0x7f6d7c0f9160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6d6c009fb0 tx=0x7f6d6c009b30).stop 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- 172.21.15.112:0/1249763577 shutdown_connections 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 --2- 172.21.15.112:0/1249763577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c060360 0x7f6d7c0f9160 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- 172.21.15.112:0/1249763577 >> 172.21.15.112:0/1249763577 conn(0x7f6d7c0f0be0 msgr2=0x7f6d7c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- 172.21.15.112:0/1249763577 shutdown_connections 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- 172.21.15.112:0/1249763577 wait complete. 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 Processor -- start 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- start start 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.986+0000 7f6d81199700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c107180 con 0x7f6d7c106870 2023-11-16T14:06:25.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60602/0 (socket says 172.21.15.112:60602) 2023-11-16T14:06:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d7ad9d700 1 -- 172.21.15.112:0/1574256462 learned_addr learned my addr 172.21.15.112:0/1574256462 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d7ad9d700 1 -- 172.21.15.112:0/1574256462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c005040 con 0x7f6d7c106870 2023-11-16T14:06:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d7ad9d700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6d6c009880 tx=0x7f6d6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6c013680 con 0x7f6d7c106870 2023-11-16T14:06:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c107380 con 0x7f6d7c106870 2023-11-16T14:06:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6d6c0095f0 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6c014e40 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.987+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c107ed0 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.988+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6d6c014420 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.988+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c0591e0 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.988+0000 7f6d637fe700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 0x7f6d64033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.988+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6d6c042540 con 0x7f6d7c106870 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.989+0000 7f6d7a59c700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 0x7f6d64033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:25.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.989+0000 7f6d7a59c700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 0x7f6d64033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6d700096a0 tx=0x7f6d70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:25.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:25.993+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6d6c025cc0 con 0x7f6d7c106870 2023-11-16T14:06:26.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.197+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d7c108690 con 0x7f6d7c106870 2023-11-16T14:06:26.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.198+0000 7f6d637fe700 1 -- 172.21.15.112:0/1574256462 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d6c025ae0 con 0x7f6d7c106870 2023-11-16T14:06:26.201 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:26.201 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 msgr2=0x7f6d64033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 0x7f6d64033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6d700096a0 tx=0x7f6d70006b40).stop 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 msgr2=0x7f6d7c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6d6c009880 tx=0x7f6d6c009b30).stop 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 shutdown_connections 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6d64031090 0x7f6d64033550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:26.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 --2- 172.21.15.112:0/1574256462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7c106870 0x7f6d7c106c90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:26.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 >> 172.21.15.112:0/1574256462 conn(0x7f6d7c0f0be0 msgr2=0x7f6d7c0f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:26.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 shutdown_connections 2023-11-16T14:06:26.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:26.200+0000 7f6d81199700 1 -- 172.21.15.112:0/1574256462 wait complete. 2023-11-16T14:06:26.204 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:24.882886+0000 mgr.smithi061.zcjido (mgr.14156) 147 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:27.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:26 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:26.199780+0000 mon.smithi061 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.112:0/1574256462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:27.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:27.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:26.883220+0000 mgr.smithi061.zcjido (mgr.14156) 148 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 -- 172.21.15.112:0/1375461654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 msgr2=0x7f6e3c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 --2- 172.21.15.112:0/1375461654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0f58a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f6e24009fb0 tx=0x7f6e24009b30).stop 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 -- 172.21.15.112:0/1375461654 shutdown_connections 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 --2- 172.21.15.112:0/1375461654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0f58a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 -- 172.21.15.112:0/1375461654 >> 172.21.15.112:0/1375461654 conn(0x7f6e3c0f0be0 msgr2=0x7f6e3c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 -- 172.21.15.112:0/1375461654 shutdown_connections 2023-11-16T14:06:28.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 -- 172.21.15.112:0/1375461654 wait complete. 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.672+0000 7f6e42948700 1 Processor -- start 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e42948700 1 -- start start 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e42948700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e42948700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0fb8e0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60620/0 (socket says 172.21.15.112:60620) 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e3bfff700 1 -- 172.21.15.112:0/3182523512 learned_addr learned my addr 172.21.15.112:0/3182523512 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.673+0000 7f6e3bfff700 1 -- 172.21.15.112:0/3182523512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24005040 con 0x7f6e3c0f5480 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e3bfff700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f6e240089f0 tx=0x7f6e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:28.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e24009320 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c0fba80 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6e24013ec0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e24014d30 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3c0fbf10 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.674+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6e240263a0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.675+0000 7f6e38ff9700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 0x7f6e28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:28.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.675+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6e240423e0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.675+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3c0591e0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.675+0000 7f6e3b7fe700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 0x7f6e28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:28.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.675+0000 7f6e3b7fe700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 0x7f6e28033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6e2c0096a0 tx=0x7f6e2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:28.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.680+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6e240266d0 con 0x7f6e3c0f5480 2023-11-16T14:06:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.888+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e3c0fc610 con 0x7f6e3c0f5480 2023-11-16T14:06:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.889+0000 7f6e38ff9700 1 -- 172.21.15.112:0/3182523512 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e2402b020 con 0x7f6e3c0f5480 2023-11-16T14:06:28.892 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:28.892 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:28.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.890+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 msgr2=0x7f6e28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:28.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.890+0000 7f6e42948700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 0x7f6e28033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6e2c0096a0 tx=0x7f6e2c006b40).stop 2023-11-16T14:06:28.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 msgr2=0x7f6e3c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f6e240089f0 tx=0x7f6e2400b040).stop 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 shutdown_connections 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6e28031090 0x7f6e28033550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 --2- 172.21.15.112:0/3182523512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e3c0f5480 0x7f6e3c0fd460 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 >> 172.21.15.112:0/3182523512 conn(0x7f6e3c0f0be0 msgr2=0x7f6e3c05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 shutdown_connections 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:28.891+0000 7f6e42948700 1 -- 172.21.15.112:0/3182523512 wait complete. 2023-11-16T14:06:28.894 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:29.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:28.890372+0000 mon.smithi061 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.112:0/3182523512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:28.883454+0000 mgr.smithi061.zcjido (mgr.14156) 149 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:30.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:30.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:31.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3927500361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 msgr2=0x7fdd080f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3927500361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f9160 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fdcf8009fb0 tx=0x7fdcf8009b30).stop 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3927500361 shutdown_connections 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3927500361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f9160 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3927500361 >> 172.21.15.112:0/3927500361 conn(0x7fdd080f0be0 msgr2=0x7fdd080f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3927500361 shutdown_connections 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3927500361 wait complete. 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.302+0000 7fdd0cc49700 1 Processor -- start 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0cc49700 1 -- start start 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0cc49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0cc49700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080f5060 con 0x7fdd08060360 2023-11-16T14:06:31.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60634/0 (socket says 172.21.15.112:60634) 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0659c700 1 -- 172.21.15.112:0/3446344374 learned_addr learned my addr 172.21.15.112:0/3446344374 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0659c700 1 -- 172.21.15.112:0/3446344374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf8005040 con 0x7fdd08060360 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0659c700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fdcf8009f80 tx=0x7fdcf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf8009320 con 0x7fdd08060360 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd080f5200 con 0x7fdd08060360 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdcf8013680 con 0x7fdd08060360 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.303+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf801eda0 con 0x7fdd08060360 2023-11-16T14:06:31.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.304+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd080f5690 con 0x7fdd08060360 2023-11-16T14:06:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.304+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdcf801e420 con 0x7fdd08060360 2023-11-16T14:06:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.304+0000 7fdcf6ffd700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 0x7fdcf0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.304+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd080591e0 con 0x7fdd08060360 2023-11-16T14:06:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.305+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdcf8042420 con 0x7fdd08060360 2023-11-16T14:06:31.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.305+0000 7fdd05d9b700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 0x7fdcf0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:31.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.305+0000 7fdd05d9b700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 0x7fdcf0033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fdcfc0096a0 tx=0x7fdcfc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:31.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.309+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdcf801d380 con 0x7fdd08060360 2023-11-16T14:06:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.520+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd08045bb0 con 0x7fdd08060360 2023-11-16T14:06:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.521+0000 7fdcf6ffd700 1 -- 172.21.15.112:0/3446344374 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcf8004490 con 0x7fdd08060360 2023-11-16T14:06:31.523 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:31.524 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:31.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 msgr2=0x7fdcf0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 0x7fdcf0033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fdcfc0096a0 tx=0x7fdcfc006b40).stop 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 msgr2=0x7fdd080f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fdcf8009f80 tx=0x7fdcf800b040).stop 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 shutdown_connections 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdcf0031090 0x7fdcf0033550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 --2- 172.21.15.112:0/3446344374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd08060360 0x7fdd080f6be0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 >> 172.21.15.112:0/3446344374 conn(0x7fdd080f0be0 msgr2=0x7fdd080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 shutdown_connections 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:31.522+0000 7fdd0cc49700 1 -- 172.21.15.112:0/3446344374 wait complete. 2023-11-16T14:06:31.526 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:30.883798+0000 mgr.smithi061.zcjido (mgr.14156) 150 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:31.522164+0000 mon.smithi061 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.112:0/3446344374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:32.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:32.889 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 -- 172.21.15.112:0/1664394846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 msgr2=0x7f70340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 --2- 172.21.15.112:0/1664394846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70340f58a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7030009fb0 tx=0x7f7030009b30).stop 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 -- 172.21.15.112:0/1664394846 shutdown_connections 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 --2- 172.21.15.112:0/1664394846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70340f58a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 -- 172.21.15.112:0/1664394846 >> 172.21.15.112:0/1664394846 conn(0x7f70340f0be0 msgr2=0x7f70340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 -- 172.21.15.112:0/1664394846 shutdown_connections 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.950+0000 7f703b8de700 1 -- 172.21.15.112:0/1664394846 wait complete. 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703b8de700 1 Processor -- start 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703b8de700 1 -- start start 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703b8de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:33.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703b8de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7034108ea0 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703967a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703967a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:60654/0 (socket says 172.21.15.112:60654) 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703967a700 1 -- 172.21.15.112:0/1154278652 learned_addr learned my addr 172.21.15.112:0/1154278652 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.951+0000 7f703967a700 1 -- 172.21.15.112:0/1154278652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030005040 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f703967a700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f70300089f0 tx=0x7f703000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030009320 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70341090a0 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7030013ec0 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030014d30 con 0x7f70340f5480 2023-11-16T14:06:33.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.952+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7034109530 con 0x7f70340f5480 2023-11-16T14:06:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.953+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f70300095f0 con 0x7f70340f5480 2023-11-16T14:06:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.953+0000 7f7029ffb700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 0x7f70200335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.953+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7034045bb0 con 0x7f70340f5480 2023-11-16T14:06:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.953+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f70300424a0 con 0x7f70340f5480 2023-11-16T14:06:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.953+0000 7f7038e79700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 0x7f70200335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:33.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.954+0000 7f7038e79700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 0x7f70200335a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f70240096a0 tx=0x7f7024006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:33.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:33.958+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f703001d7f0 con 0x7f70340f5480 2023-11-16T14:06:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:32.884191+0000 mgr.smithi061.zcjido (mgr.14156) 151 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:34.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.168+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70340fb060 con 0x7f70340f5480 2023-11-16T14:06:34.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.169+0000 7f7029ffb700 1 -- 172.21.15.112:0/1154278652 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f703001d7f0 con 0x7f70340f5480 2023-11-16T14:06:34.182 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:34.182 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 msgr2=0x7f70200335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 0x7f70200335a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f70240096a0 tx=0x7f7024006b40).stop 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 msgr2=0x7f70341089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f70300089f0 tx=0x7f703000b040).stop 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 shutdown_connections 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f70200310e0 0x7f70200335a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:34.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 --2- 172.21.15.112:0/1154278652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340f5480 0x7f70341089b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:34.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 >> 172.21.15.112:0/1154278652 conn(0x7f70340f0be0 msgr2=0x7f703405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:34.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 shutdown_connections 2023-11-16T14:06:34.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:34.171+0000 7f703b8de700 1 -- 172.21.15.112:0/1154278652 wait complete. 2023-11-16T14:06:34.184 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:35.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:34.170234+0000 mon.smithi061 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.112:0/1154278652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:35.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:35.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:34.884507+0000 mgr.smithi061.zcjido (mgr.14156) 152 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:34.982218+0000 mon.smithi061 (mon.0) 212 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:06:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:34.982885+0000 mon.smithi061 (mon.0) 213 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 -- 172.21.15.112:0/3220624677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 msgr2=0x7f9cc00f5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 --2- 172.21.15.112:0/3220624677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc00f5260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9cb0009fb0 tx=0x7f9cb0009b30).stop 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 -- 172.21.15.112:0/3220624677 shutdown_connections 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 --2- 172.21.15.112:0/3220624677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc00f5260 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 -- 172.21.15.112:0/3220624677 >> 172.21.15.112:0/3220624677 conn(0x7f9cc00eff20 msgr2=0x7f9cc00f2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 -- 172.21.15.112:0/3220624677 shutdown_connections 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.622+0000 7f9cc8461700 1 -- 172.21.15.112:0/3220624677 wait complete. 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc8461700 1 Processor -- start 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc8461700 1 -- start start 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc8461700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc8461700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc0106cd0 con 0x7f9cc00f4e20 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc61fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc61fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34146/0 (socket says 172.21.15.112:34146) 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.623+0000 7f9cc61fd700 1 -- 172.21.15.112:0/380918834 learned_addr learned my addr 172.21.15.112:0/380918834 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:36.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cc61fd700 1 -- 172.21.15.112:0/380918834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0005040 con 0x7f9cc00f4e20 2023-11-16T14:06:36.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cc61fd700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9cb00089f0 tx=0x7f9cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:36.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb0009320 con 0x7f9cc00f4e20 2023-11-16T14:06:36.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc0106ed0 con 0x7f9cc00f4e20 2023-11-16T14:06:36.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9cb0013ec0 con 0x7f9cc00f4e20 2023-11-16T14:06:36.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb0014d30 con 0x7f9cc00f4e20 2023-11-16T14:06:36.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.624+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc0107360 con 0x7f9cc00f4e20 2023-11-16T14:06:36.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9cb0014420 con 0x7f9cc00f4e20 2023-11-16T14:06:36.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cb6ffd700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 0x7f9cac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:36.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0045bb0 con 0x7f9cc00f4e20 2023-11-16T14:06:36.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9cb0042400 con 0x7f9cc00f4e20 2023-11-16T14:06:36.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cc59fc700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 0x7f9cac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:36.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.625+0000 7f9cc59fc700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 0x7f9cac033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9cbc0096a0 tx=0x7f9cbc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:36.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.630+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9cb001d980 con 0x7f9cc00f4e20 2023-11-16T14:06:36.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.839+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cc00fa680 con 0x7f9cc00f4e20 2023-11-16T14:06:36.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.839+0000 7f9cb6ffd700 1 -- 172.21.15.112:0/380918834 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9cb001d380 con 0x7f9cc00f4e20 2023-11-16T14:06:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:36.843 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 msgr2=0x7f9cac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 0x7f9cac033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9cbc0096a0 tx=0x7f9cbc006b40).stop 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 msgr2=0x7f9cc01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9cb00089f0 tx=0x7f9cb000b040).stop 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 shutdown_connections 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9cac031090 0x7f9cac033550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 --2- 172.21.15.112:0/380918834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc00f4e20 0x7f9cc01067e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 >> 172.21.15.112:0/380918834 conn(0x7f9cc00eff20 msgr2=0x7f9cc00f1110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 shutdown_connections 2023-11-16T14:06:36.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:36.841+0000 7f9cc8461700 1 -- 172.21.15.112:0/380918834 wait complete. 2023-11-16T14:06:36.845 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:37.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:36.840943+0000 mon.smithi061 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.112:0/380918834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:38.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:38.216 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:36.884880+0000 mgr.smithi061.zcjido (mgr.14156) 153 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:39.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 -- 172.21.15.112:0/288925190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28840f6590 msgr2=0x7f28840f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:39.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 --2- 172.21.15.112:0/288925190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28840f6590 0x7f28840f69b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2874004760 tx=0x7f2874009ad0).stop 2023-11-16T14:06:39.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 -- 172.21.15.112:0/288925190 shutdown_connections 2023-11-16T14:06:39.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 --2- 172.21.15.112:0/288925190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28840f6590 0x7f28840f69b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:39.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 -- 172.21.15.112:0/288925190 >> 172.21.15.112:0/288925190 conn(0x7f28840f1cd0 msgr2=0x7f28840f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:39.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 -- 172.21.15.112:0/288925190 shutdown_connections 2023-11-16T14:06:39.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.316+0000 7f288b087700 1 -- 172.21.15.112:0/288925190 wait complete. 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.317+0000 7f288b087700 1 Processor -- start 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.317+0000 7f288b087700 1 -- start start 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.317+0000 7f288b087700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.317+0000 7f288b087700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884107240 con 0x7f2884106930 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f2888e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f2888e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34162/0 (socket says 172.21.15.112:34162) 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f2888e23700 1 -- 172.21.15.112:0/3274066177 learned_addr learned my addr 172.21.15.112:0/3274066177 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f2888e23700 1 -- 172.21.15.112:0/3274066177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874009d70 con 0x7f2884106930 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f2888e23700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f2874009820 tx=0x7f2874009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f287400e040 con 0x7f2884106930 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f28740095b0 con 0x7f2884106930 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2884107440 con 0x7f2884106930 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2874014be0 con 0x7f2884106930 2023-11-16T14:06:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.318+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2884107f40 con 0x7f2884106930 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.319+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f28740283a0 con 0x7f2884106930 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.319+0000 7f28817fa700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 0x7f286c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.319+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28840591e0 con 0x7f2884106930 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.319+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2874042220 con 0x7f2884106930 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.320+0000 7f2883fff700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 0x7f286c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:39.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.320+0000 7f2883fff700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 0x7f286c0335a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f28780096a0 tx=0x7f2878006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:39.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.324+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f28740262b0 con 0x7f2884106930 2023-11-16T14:06:39.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.536+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2884045bb0 con 0x7f2884106930 2023-11-16T14:06:39.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.536+0000 7f28817fa700 1 -- 172.21.15.112:0/3274066177 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f287401eb60 con 0x7f2884106930 2023-11-16T14:06:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:39.539 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 msgr2=0x7f286c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 0x7f286c0335a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f28780096a0 tx=0x7f2878006b40).stop 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 msgr2=0x7f2884106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f2874009820 tx=0x7f2874009ad0).stop 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 shutdown_connections 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f286c0310e0 0x7f286c0335a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 --2- 172.21.15.112:0/3274066177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2884106930 0x7f2884106d50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 >> 172.21.15.112:0/3274066177 conn(0x7f28840f1cd0 msgr2=0x7f28840f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:39.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 shutdown_connections 2023-11-16T14:06:39.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:39.538+0000 7f288b087700 1 -- 172.21.15.112:0/3274066177 wait complete. 2023-11-16T14:06:39.542 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:38.885188+0000 mgr.smithi061.zcjido (mgr.14156) 154 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:39.537856+0000 mon.smithi061 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.112:0/3274066177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:40.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:40.912 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:41.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 -- 172.21.15.112:0/3448801092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 msgr2=0x7fde680f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:41.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 --2- 172.21.15.112:0/3448801092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde680f69b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fde58004760 tx=0x7fde58009ad0).stop 2023-11-16T14:06:41.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 -- 172.21.15.112:0/3448801092 shutdown_connections 2023-11-16T14:06:41.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 --2- 172.21.15.112:0/3448801092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde680f69b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 -- 172.21.15.112:0/3448801092 >> 172.21.15.112:0/3448801092 conn(0x7fde680f1cd0 msgr2=0x7fde680f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 -- 172.21.15.112:0/3448801092 shutdown_connections 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 -- 172.21.15.112:0/3448801092 wait complete. 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.956+0000 7fde6e896700 1 Processor -- start 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde6e896700 1 -- start start 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde6e896700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde6e896700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68106cb0 con 0x7fde680f6590 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34184/0 (socket says 172.21.15.112:34184) 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde67fff700 1 -- 172.21.15.112:0/380971977 learned_addr learned my addr 172.21.15.112:0/380971977 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde67fff700 1 -- 172.21.15.112:0/380971977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58009d70 con 0x7fde680f6590 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde67fff700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fde58004170 tx=0x7fde58009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:41.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.957+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde5800e040 con 0x7fde680f6590 2023-11-16T14:06:41.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68106eb0 con 0x7fde680f6590 2023-11-16T14:06:41.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fde58008ef0 con 0x7fde680f6590 2023-11-16T14:06:41.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde58014c90 con 0x7fde680f6590 2023-11-16T14:06:41.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68107340 con 0x7fde680f6590 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fde58014420 con 0x7fde680f6590 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde64ff9700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.958+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68045bb0 con 0x7fde680f6590 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.959+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fde58042210 con 0x7fde680f6590 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.959+0000 7fde677fe700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:41.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.959+0000 7fde677fe700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 0x7fde50033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:41.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:41.964+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde58026300 con 0x7fde680f6590 2023-11-16T14:06:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.171+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde68108190 con 0x7fde680f6590 2023-11-16T14:06:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.171+0000 7fde64ff9700 1 -- 172.21.15.112:0/380971977 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde580210c0 con 0x7fde680f6590 2023-11-16T14:06:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:42.174 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 msgr2=0x7fde50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 0x7fde50033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).stop 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 msgr2=0x7fde681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fde58004170 tx=0x7fde58009ad0).stop 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 shutdown_connections 2023-11-16T14:06:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fde50031090 0x7fde50033550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 --2- 172.21.15.112:0/380971977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680f6590 0x7fde681067c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 >> 172.21.15.112:0/380971977 conn(0x7fde680f1cd0 msgr2=0x7fde680f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 shutdown_connections 2023-11-16T14:06:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:42.173+0000 7fde6e896700 1 -- 172.21.15.112:0/380971977 wait complete. 2023-11-16T14:06:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:40.885506+0000 mgr.smithi061.zcjido (mgr.14156) 155 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:43.194 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:42.172932+0000 mon.smithi061 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.112:0/380971977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:43.195 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:42.836378+0000 mon.smithi061 (mon.0) 217 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi061", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-16T14:06:43.195 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:42.839110+0000 mon.smithi061 (mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi061", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-16T14:06:43.195 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:42 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:42.839940+0000 mon.smithi061 (mon.0) 219 : audit [DBG] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T14:06:43.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:43.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:06:42.834298+0000 mgr.smithi061.zcjido (mgr.14156) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon alertmanager.smithi061 ... 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:59Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:03:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.165 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:04:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:05:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:06:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:06:44.166 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('uid/gid not found') 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: uid/gid not found 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: yield (conn, connr) 2023-11-16T14:06:44.167 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: code, '\n'.join(err))) 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon alertmanager.smithi061 ... 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:59Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:03:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.168 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:04:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:05:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:06:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-16T14:06:44.169 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('uid/gid not found') 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: uid/gid not found 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:06:42.834574+0000 mgr.smithi061.zcjido (mgr.14156) 157 : cephadm [ERR] Failed while placing alertmanager.smithi061 on smithi061: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-alertmanager.smithi061 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon alertmanager.smithi061 ... 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:02:59Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.170 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:03:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:04:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:05:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr time="2023-11-16T14:06:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:06:44.171 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('uid/gid not found') 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: uid/gid not found 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:06:42.840708+0000 mgr.smithi061.zcjido (mgr.14156) 158 : cephadm [INF] Deploying daemon crash.smithi061 on smithi061 2023-11-16T14:06:44.172 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:42.885761+0000 mgr.smithi061.zcjido (mgr.14156) 159 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:44.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 -- 172.21.15.112:0/1976764249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 msgr2=0x7f502c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 --2- 172.21.15.112:0/1976764249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c0f9160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f501c009fb0 tx=0x7f501c009b30).stop 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 -- 172.21.15.112:0/1976764249 shutdown_connections 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 --2- 172.21.15.112:0/1976764249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c0f9160 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 -- 172.21.15.112:0/1976764249 >> 172.21.15.112:0/1976764249 conn(0x7f502c0f0be0 msgr2=0x7f502c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 -- 172.21.15.112:0/1976764249 shutdown_connections 2023-11-16T14:06:44.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.630+0000 7f5034456700 1 -- 172.21.15.112:0/1976764249 wait complete. 2023-11-16T14:06:44.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f5034456700 1 Processor -- start 2023-11-16T14:06:44.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f5034456700 1 -- start start 2023-11-16T14:06:44.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f5034456700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:44.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f5034456700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c106c60 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f50321f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f50321f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34190/0 (socket says 172.21.15.112:34190) 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f50321f2700 1 -- 172.21.15.112:0/2056450414 learned_addr learned my addr 172.21.15.112:0/2056450414 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f50321f2700 1 -- 172.21.15.112:0/2056450414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c005040 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f50321f2700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f501c009f80 tx=0x7f501c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.631+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f501c009320 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f501c013680 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f501c01eda0 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c106e60 con 0x7f502c060360 2023-11-16T14:06:44.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c1072f0 con 0x7f502c060360 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f501c014420 con 0x7f502c060360 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5022ffd700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 0x7f5018033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f501c0423e0 con 0x7f502c060360 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.632+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c045bb0 con 0x7f502c060360 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.633+0000 7f50319f1700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 0x7f5018033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.633+0000 7f50319f1700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 0x7f5018033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f50280096a0 tx=0x7f5028006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:44.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.637+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f501c01e420 con 0x7f502c060360 2023-11-16T14:06:44.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.844+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f502c0fc150 con 0x7f502c060360 2023-11-16T14:06:44.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.844+0000 7f5022ffd700 1 -- 172.21.15.112:0/2056450414 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f501c01cb10 con 0x7f502c060360 2023-11-16T14:06:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:44.847 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 msgr2=0x7f5018033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 0x7f5018033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f50280096a0 tx=0x7f5028006b40).stop 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 msgr2=0x7f502c106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f501c009f80 tx=0x7f501c00b040).stop 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 shutdown_connections 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5018031090 0x7f5018033550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 --2- 172.21.15.112:0/2056450414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f502c060360 0x7f502c106770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:44.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 >> 172.21.15.112:0/2056450414 conn(0x7f502c0f0be0 msgr2=0x7f502c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:44.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 shutdown_connections 2023-11-16T14:06:44.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:44.846+0000 7f5034456700 1 -- 172.21.15.112:0/2056450414 wait complete. 2023-11-16T14:06:44.849 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.347096+0000 mon.smithi061 (mon.0) 220 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.517056+0000 mon.smithi061 (mon.0) 221 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.519678+0000 mon.smithi061 (mon.0) 222 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.520044+0000 mon.smithi061 (mon.0) 223 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.523469+0000 mon.smithi061 (mon.0) 224 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.845115+0000 mon.smithi061 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.112:0/2056450414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:45.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.980265+0000 mon.smithi061 (mon.0) 226 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:06:46.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:46.214 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:46.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:46 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:44.520364+0000 mgr.smithi061.zcjido (mgr.14156) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-16T14:06:46.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:46 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:06:44.526596+0000 mgr.smithi061.zcjido (mgr.14156) 161 : cephadm [INF] Deploying daemon grafana.smithi061 on smithi061 2023-11-16T14:06:46.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:46 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:44.886133+0000 mgr.smithi061.zcjido (mgr.14156) 162 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- 172.21.15.112:0/167121577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 msgr2=0x7fb6400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 --2- 172.21.15.112:0/167121577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6400f58a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb634009fb0 tx=0x7fb634009b30).stop 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- 172.21.15.112:0/167121577 shutdown_connections 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 --2- 172.21.15.112:0/167121577 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6400f58a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- 172.21.15.112:0/167121577 >> 172.21.15.112:0/167121577 conn(0x7fb6400f0be0 msgr2=0x7fb6400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- 172.21.15.112:0/167121577 shutdown_connections 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- 172.21.15.112:0/167121577 wait complete. 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 Processor -- start 2023-11-16T14:06:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.280+0000 7fb646cc5700 1 -- start start 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb646cc5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb646cc5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640105be0 con 0x7fb6400f5480 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb644a61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb644a61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:33470/0 (socket says 172.21.15.112:33470) 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb644a61700 1 -- 172.21.15.112:0/1028158978 learned_addr learned my addr 172.21.15.112:0/1028158978 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.281+0000 7fb644a61700 1 -- 172.21.15.112:0/1028158978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634005040 con 0x7fb6400f5480 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb644a61700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb6340089f0 tx=0x7fb63400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb634009320 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb634013ec0 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb640105de0 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb634014d30 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.282+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb640106270 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb6340263a0 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb63d7fa700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 0x7fb630033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb6340423d0 con 0x7fb6400f5480 2023-11-16T14:06:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb640045bb0 con 0x7fb6400f5480 2023-11-16T14:06:47.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb63ffff700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 0x7fb630033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:47.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.283+0000 7fb63ffff700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 0x7fb630033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb62c0096a0 tx=0x7fb62c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.288+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb6340266d0 con 0x7fb6400f5480 2023-11-16T14:06:47.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.494+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6400fb060 con 0x7fb6400f5480 2023-11-16T14:06:47.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.495+0000 7fb63d7fa700 1 -- 172.21.15.112:0/1028158978 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb63402b020 con 0x7fb6400f5480 2023-11-16T14:06:47.498 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:47.498 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 msgr2=0x7fb630033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 0x7fb630033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb62c0096a0 tx=0x7fb62c006b40).stop 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 msgr2=0x7fb6401056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb6340089f0 tx=0x7fb63400b040).stop 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 shutdown_connections 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb630031090 0x7fb630033550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 --2- 172.21.15.112:0/1028158978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb6400f5480 0x7fb6401056f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:47.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 >> 172.21.15.112:0/1028158978 conn(0x7fb6400f0be0 msgr2=0x7fb6400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:47.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 shutdown_connections 2023-11-16T14:06:47.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:47.497+0000 7fb646cc5700 1 -- 172.21.15.112:0/1028158978 wait complete. 2023-11-16T14:06:47.501 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:48.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:46.886376+0000 mgr.smithi061.zcjido (mgr.14156) 163 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:48.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:47.495705+0000 mon.smithi061 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.112:0/1028158978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:48.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:48.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:49.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 -- 172.21.15.112:0/283548589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 msgr2=0x7f7d940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:49.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/283548589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d940f58a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f7d84009fb0 tx=0x7f7d84009b30).stop 2023-11-16T14:06:49.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 -- 172.21.15.112:0/283548589 shutdown_connections 2023-11-16T14:06:49.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/283548589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d940f58a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:49.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 -- 172.21.15.112:0/283548589 >> 172.21.15.112:0/283548589 conn(0x7f7d940f0be0 msgr2=0x7f7d940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 -- 172.21.15.112:0/283548589 shutdown_connections 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.941+0000 7f7d98a2e700 1 -- 172.21.15.112:0/283548589 wait complete. 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d98a2e700 1 Processor -- start 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d98a2e700 1 -- start start 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d98a2e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d98a2e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94060300 con 0x7f7d940f5480 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d9259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d9259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:33474/0 (socket says 172.21.15.112:33474) 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d9259c700 1 -- 172.21.15.112:0/2467124045 learned_addr learned my addr 172.21.15.112:0/2467124045 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d9259c700 1 -- 172.21.15.112:0/2467124045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84005040 con 0x7f7d940f5480 2023-11-16T14:06:49.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.942+0000 7f7d9259c700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7d840089f0 tx=0x7f7d8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.943+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d84009320 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.943+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94060500 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.943+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7d84013ec0 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.943+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d84014d30 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.943+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d94109bc0 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7d840095f0 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d940591e0 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d82ffd700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 0x7f7d7c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7d840424e0 con 0x7f7d940f5480 2023-11-16T14:06:49.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d91d9b700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 0x7f7d7c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:49.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.944+0000 7f7d91d9b700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 0x7f7d7c033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f7d880096a0 tx=0x7f7d88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:49.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:49.949+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7d8401d7f0 con 0x7f7d940f5480 2023-11-16T14:06:50.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.156+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d94045bb0 con 0x7f7d940f5480 2023-11-16T14:06:50.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.157+0000 7f7d82ffd700 1 -- 172.21.15.112:0/2467124045 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d8400fe40 con 0x7f7d940f5480 2023-11-16T14:06:50.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:50.159 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 msgr2=0x7f7d7c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 0x7f7d7c033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f7d880096a0 tx=0x7f7d88006b40).stop 2023-11-16T14:06:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 msgr2=0x7f7d94109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7d840089f0 tx=0x7f7d8400b040).stop 2023-11-16T14:06:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 shutdown_connections 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7d7c031090 0x7f7d7c033550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 --2- 172.21.15.112:0/2467124045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d940f5480 0x7f7d94109510 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 >> 172.21.15.112:0/2467124045 conn(0x7f7d940f0be0 msgr2=0x7f7d940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 shutdown_connections 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:50.158+0000 7f7d98a2e700 1 -- 172.21.15.112:0/2467124045 wait complete. 2023-11-16T14:06:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:50.758 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:48.886734+0000 mgr.smithi061.zcjido (mgr.14156) 164 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:50.759 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:50.157267+0000 mon.smithi061 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.112:0/2467124045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:51.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:51.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:52.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 -- 172.21.15.112:0/3321065201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 msgr2=0x7f12a80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:52.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 --2- 172.21.15.112:0/3321065201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a80f58d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1298009fb0 tx=0x7f1298009b30).stop 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 -- 172.21.15.112:0/3321065201 shutdown_connections 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 --2- 172.21.15.112:0/3321065201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a80f58d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 -- 172.21.15.112:0/3321065201 >> 172.21.15.112:0/3321065201 conn(0x7f12a80f0c10 msgr2=0x7f12a80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 -- 172.21.15.112:0/3321065201 shutdown_connections 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.637+0000 7f12ad768700 1 -- 172.21.15.112:0/3321065201 wait complete. 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12ad768700 1 Processor -- start 2023-11-16T14:06:52.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12ad768700 1 -- start start 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12ad768700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12ad768700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8060360 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:33490/0 (socket says 172.21.15.112:33490) 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12a6ffd700 1 -- 172.21.15.112:0/301340479 learned_addr learned my addr 172.21.15.112:0/301340479 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12a6ffd700 1 -- 172.21.15.112:0/301340479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298005040 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12a6ffd700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f1298000c00 tx=0x7f129800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12980095f0 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1298007e50 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a8060560 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.638+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1298014e70 con 0x7f12a80f54b0 2023-11-16T14:06:52.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.639+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8109b90 con 0x7f12a80f54b0 2023-11-16T14:06:52.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.639+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1298025070 con 0x7f12a80f54b0 2023-11-16T14:06:52.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.639+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8045bb0 con 0x7f12a80f54b0 2023-11-16T14:06:52.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.639+0000 7f128f7fe700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 0x7f12900335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:52.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.640+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1298041e20 con 0x7f12a80f54b0 2023-11-16T14:06:52.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.640+0000 7f12a67fc700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 0x7f12900335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:52.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.640+0000 7f12a67fc700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 0x7f12900335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f129c0096a0 tx=0x7f129c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:52.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.645+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1298046070 con 0x7f12a80f54b0 2023-11-16T14:06:52.757 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:50.887155+0000 mgr.smithi061.zcjido (mgr.14156) 165 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:52.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.853+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12a810a3b0 con 0x7f12a80f54b0 2023-11-16T14:06:52.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.854+0000 7f128f7fe700 1 -- 172.21.15.112:0/301340479 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1298025e20 con 0x7f12a80f54b0 2023-11-16T14:06:52.856 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:52.856 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:52.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 msgr2=0x7f12900335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:52.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 0x7f12900335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f129c0096a0 tx=0x7f129c006b40).stop 2023-11-16T14:06:52.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 msgr2=0x7f12a8109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:52.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f1298000c00 tx=0x7f129800b040).stop 2023-11-16T14:06:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 shutdown_connections 2023-11-16T14:06:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f12900310e0 0x7f12900335a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 --2- 172.21.15.112:0/301340479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a80f54b0 0x7f12a8109460 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 >> 172.21.15.112:0/301340479 conn(0x7f12a80f0c10 msgr2=0x7f12a80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 shutdown_connections 2023-11-16T14:06:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:52.855+0000 7f12ad768700 1 -- 172.21.15.112:0/301340479 wait complete. 2023-11-16T14:06:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:52.854128+0000 mon.smithi061 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.112:0/301340479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:52.887490+0000 mgr.smithi061.zcjido (mgr.14156) 166 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:54.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:54.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.330+0000 7fa51d841700 1 -- 172.21.15.112:0/2116458803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 msgr2=0x7fa5180f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.330+0000 7fa51d841700 1 --2- 172.21.15.112:0/2116458803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5180f58d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fa508009fb0 tx=0x7fa508009b30).stop 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.330+0000 7fa51d841700 1 -- 172.21.15.112:0/2116458803 shutdown_connections 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.330+0000 7fa51d841700 1 --2- 172.21.15.112:0/2116458803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5180f58d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.330+0000 7fa51d841700 1 -- 172.21.15.112:0/2116458803 >> 172.21.15.112:0/2116458803 conn(0x7fa5180f0c10 msgr2=0x7fa5180f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.331+0000 7fa51d841700 1 -- 172.21.15.112:0/2116458803 shutdown_connections 2023-11-16T14:06:55.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.331+0000 7fa51d841700 1 -- 172.21.15.112:0/2116458803 wait complete. 2023-11-16T14:06:55.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.331+0000 7fa51d841700 1 Processor -- start 2023-11-16T14:06:55.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.331+0000 7fa51d841700 1 -- start start 2023-11-16T14:06:55.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa51d841700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa51d841700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518105b90 con 0x7fa5180f54b0 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa516ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa516ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40654/0 (socket says 172.21.15.112:40654) 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa516ffd700 1 -- 172.21.15.112:0/996630203 learned_addr learned my addr 172.21.15.112:0/996630203 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa516ffd700 1 -- 172.21.15.112:0/996630203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508005040 con 0x7fa5180f54b0 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa516ffd700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fa5080089f0 tx=0x7fa50800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa508009320 con 0x7fa5180f54b0 2023-11-16T14:06:55.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa508013ec0 con 0x7fa5180f54b0 2023-11-16T14:06:55.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa508014d30 con 0x7fa5180f54b0 2023-11-16T14:06:55.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.332+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518105d90 con 0x7fa5180f54b0 2023-11-16T14:06:55.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.333+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518106220 con 0x7fa5180f54b0 2023-11-16T14:06:55.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.333+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa5080263a0 con 0x7fa5180f54b0 2023-11-16T14:06:55.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.333+0000 7fa4ff7fe700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 0x7fa500033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:55.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.333+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa508042450 con 0x7fa5180f54b0 2023-11-16T14:06:55.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.333+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa518045bb0 con 0x7fa5180f54b0 2023-11-16T14:06:55.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.334+0000 7fa5167fc700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 0x7fa500033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:55.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.334+0000 7fa5167fc700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 0x7fa500033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa50c0096a0 tx=0x7fa50c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:55.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.338+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa5080266d0 con 0x7fa5180f54b0 2023-11-16T14:06:55.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.548+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5180fb090 con 0x7fa5180f54b0 2023-11-16T14:06:55.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.549+0000 7fa4ff7fe700 1 -- 172.21.15.112:0/996630203 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa50802b020 con 0x7fa5180f54b0 2023-11-16T14:06:55.551 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:55.551 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 msgr2=0x7fa500033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 0x7fa500033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa50c0096a0 tx=0x7fa50c006b40).stop 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 msgr2=0x7fa5181056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fa5080089f0 tx=0x7fa50800b040).stop 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 shutdown_connections 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa500031090 0x7fa500033550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 --2- 172.21.15.112:0/996630203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5180f54b0 0x7fa5181056a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 >> 172.21.15.112:0/996630203 conn(0x7fa5180f0c10 msgr2=0x7fa5180f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 shutdown_connections 2023-11-16T14:06:55.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:55.551+0000 7fa51d841700 1 -- 172.21.15.112:0/996630203 wait complete. 2023-11-16T14:06:55.554 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:54.887767+0000 mgr.smithi061.zcjido (mgr.14156) 167 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:55.549637+0000 mon.smithi061 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.112:0/996630203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:56.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:56.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:06:57.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 -- 172.21.15.112:0/2153744009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 msgr2=0x7f54180f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 --2- 172.21.15.112:0/2153744009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f54180f69b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f540c004760 tx=0x7f540c009ad0).stop 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 -- 172.21.15.112:0/2153744009 shutdown_connections 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 --2- 172.21.15.112:0/2153744009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f54180f69b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 -- 172.21.15.112:0/2153744009 >> 172.21.15.112:0/2153744009 conn(0x7f54180f1cd0 msgr2=0x7f54180f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 -- 172.21.15.112:0/2153744009 shutdown_connections 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 -- 172.21.15.112:0/2153744009 wait complete. 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.973+0000 7f541cac3700 1 Processor -- start 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541cac3700 1 -- start start 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541cac3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541cac3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5418069f00 con 0x7f54180f6590 2023-11-16T14:06:57.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40676/0 (socket says 172.21.15.112:40676) 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541659c700 1 -- 172.21.15.112:0/1147634588 learned_addr learned my addr 172.21.15.112:0/1147634588 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541659c700 1 -- 172.21.15.112:0/1147634588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c009d70 con 0x7f54180f6590 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541659c700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f540c004170 tx=0x7f540c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c00e040 con 0x7f54180f6590 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541806a0a0 con 0x7f54180f6590 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f540c008ef0 con 0x7f54180f6590 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c014c90 con 0x7f54180f6590 2023-11-16T14:06:57.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.974+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541806a530 con 0x7f54180f6590 2023-11-16T14:06:57.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.975+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f540c0283f0 con 0x7f54180f6590 2023-11-16T14:06:57.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.975+0000 7f5406ffd700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 0x7f5400033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:06:57.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.976+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f540c042210 con 0x7f54180f6590 2023-11-16T14:06:57.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.976+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54180591e0 con 0x7f54180f6590 2023-11-16T14:06:57.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.976+0000 7f5415d9b700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 0x7f5400033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:06:57.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.976+0000 7f5415d9b700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 0x7f5400033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f54080096a0 tx=0x7f5408006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:06:57.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:57.981+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f540c026300 con 0x7f54180f6590 2023-11-16T14:06:58.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.190+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f541806ac90 con 0x7f54180f6590 2023-11-16T14:06:58.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.190+0000 7f5406ffd700 1 -- 172.21.15.112:0/1147634588 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f540c012b20 con 0x7f54180f6590 2023-11-16T14:06:58.192 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:06:58.192 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 msgr2=0x7f5400033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 0x7f5400033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f54080096a0 tx=0x7f5408006b40).stop 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 msgr2=0x7f541806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f540c004170 tx=0x7f540c009ad0).stop 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 shutdown_connections 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5400031090 0x7f5400033550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 --2- 172.21.15.112:0/1147634588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54180f6590 0x7f541806ba80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:06:58.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 >> 172.21.15.112:0/1147634588 conn(0x7f54180f1cd0 msgr2=0x7f54180f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:06:58.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 shutdown_connections 2023-11-16T14:06:58.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:06:58.192+0000 7f541cac3700 1 -- 172.21.15.112:0/1147634588 wait complete. 2023-11-16T14:06:58.195 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:06:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:56.888000+0000 mgr.smithi061.zcjido (mgr.14156) 168 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:06:59.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:58 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:06:58.190882+0000 mon.smithi061 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.112:0/1147634588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:06:59.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:06:59.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:00.218 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:06:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:06:58.888250+0000 mgr.smithi061.zcjido (mgr.14156) 169 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:00.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 -- 172.21.15.112:0/3553993037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 msgr2=0x7f366c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:00.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3553993037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c0f58a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3660009fb0 tx=0x7f3660009b30).stop 2023-11-16T14:07:00.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 -- 172.21.15.112:0/3553993037 shutdown_connections 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3553993037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c0f58a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 -- 172.21.15.112:0/3553993037 >> 172.21.15.112:0/3553993037 conn(0x7f366c0f0be0 msgr2=0x7f366c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 -- 172.21.15.112:0/3553993037 shutdown_connections 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.671+0000 7f3670fb6700 1 -- 172.21.15.112:0/3553993037 wait complete. 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f3670fb6700 1 Processor -- start 2023-11-16T14:07:00.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f3670fb6700 1 -- start start 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f3670fb6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f3670fb6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c105be0 con 0x7f366c0f5480 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f366a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f366a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40696/0 (socket says 172.21.15.112:40696) 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f366a59c700 1 -- 172.21.15.112:0/3815870002 learned_addr learned my addr 172.21.15.112:0/3815870002 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f366a59c700 1 -- 172.21.15.112:0/3815870002 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660005040 con 0x7f366c0f5480 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.672+0000 7f366a59c700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f36600089f0 tx=0x7f366000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.673+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3660009320 con 0x7f366c0f5480 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.673+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c105de0 con 0x7f366c0f5480 2023-11-16T14:07:00.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.673+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3660013ec0 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.673+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3660014d30 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.673+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c106270 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f36600095f0 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c045bb0 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f365affd700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 0x7f3654033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f36600424a0 con 0x7f366c0f5480 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f3669d9b700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 0x7f3654033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:00.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.674+0000 7f3669d9b700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 0x7f3654033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f365c0096a0 tx=0x7f365c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:00.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.681+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f366001d7f0 con 0x7f366c0f5480 2023-11-16T14:07:00.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.888+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f366c0faf70 con 0x7f366c0f5480 2023-11-16T14:07:00.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.889+0000 7f365affd700 1 -- 172.21.15.112:0/3815870002 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f366000fe40 con 0x7f366c0f5480 2023-11-16T14:07:00.892 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:00.892 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:00.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 msgr2=0x7f3654033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 0x7f3654033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f365c0096a0 tx=0x7f365c006b40).stop 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 msgr2=0x7f366c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f36600089f0 tx=0x7f366000b040).stop 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 shutdown_connections 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3654031090 0x7f3654033550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:00.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 --2- 172.21.15.112:0/3815870002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c0f5480 0x7f366c1056f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:00.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.890+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 >> 172.21.15.112:0/3815870002 conn(0x7f366c0f0be0 msgr2=0x7f366c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:00.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.891+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 shutdown_connections 2023-11-16T14:07:00.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:00.891+0000 7f3670fb6700 1 -- 172.21.15.112:0/3815870002 wait complete. 2023-11-16T14:07:00.896 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:01.254 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:00 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:00.890482+0000 mon.smithi061 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.112:0/3815870002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:02.065 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:00.888511+0000 mgr.smithi061.zcjido (mgr.14156) 170 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:02.065 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:01.890819+0000 mon.smithi061 (mon.0) 233 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:07:02.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:02.270 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:03.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:02 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:07:01.893108+0000 mgr.smithi061.zcjido (mgr.14156) 171 : cephadm [INF] Deploying daemon node-exporter.smithi061 on smithi061 2023-11-16T14:07:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- 172.21.15.112:0/206709532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 msgr2=0x7fe6400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 --2- 172.21.15.112:0/206709532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6400f58a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe634009fb0 tx=0x7fe634009b30).stop 2023-11-16T14:07:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- 172.21.15.112:0/206709532 shutdown_connections 2023-11-16T14:07:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 --2- 172.21.15.112:0/206709532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6400f58a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- 172.21.15.112:0/206709532 >> 172.21.15.112:0/206709532 conn(0x7fe6400f0be0 msgr2=0x7fe6400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:03.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- 172.21.15.112:0/206709532 shutdown_connections 2023-11-16T14:07:03.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- 172.21.15.112:0/206709532 wait complete. 2023-11-16T14:07:03.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 Processor -- start 2023-11-16T14:07:03.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.352+0000 7fe646c2a700 1 -- start start 2023-11-16T14:07:03.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe646c2a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:03.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe646c2a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe640060320 con 0x7fe6400f5480 2023-11-16T14:07:03.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe6449c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:03.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe6449c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:40712/0 (socket says 172.21.15.112:40712) 2023-11-16T14:07:03.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe6449c6700 1 -- 172.21.15.112:0/35291505 learned_addr learned my addr 172.21.15.112:0/35291505 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:03.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe6449c6700 1 -- 172.21.15.112:0/35291505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634005040 con 0x7fe6400f5480 2023-11-16T14:07:03.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.353+0000 7fe6449c6700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fe6340089f0 tx=0x7fe63400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:03.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe634009320 con 0x7fe6400f5480 2023-11-16T14:07:03.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe640060520 con 0x7fe6400f5480 2023-11-16T14:07:03.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe634013ec0 con 0x7fe6400f5480 2023-11-16T14:07:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe634014d30 con 0x7fe6400f5480 2023-11-16T14:07:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe640109ba0 con 0x7fe6400f5480 2023-11-16T14:07:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.354+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe6340263a0 con 0x7fe6400f5480 2023-11-16T14:07:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.355+0000 7fe63d7fa700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 0x7fe630033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.355+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe6340425c0 con 0x7fe6400f5480 2023-11-16T14:07:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.355+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6400591e0 con 0x7fe6400f5480 2023-11-16T14:07:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.355+0000 7fe63ffff700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 0x7fe630033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.355+0000 7fe63ffff700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 0x7fe630033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe62c0096a0 tx=0x7fe62c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.360+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe634012920 con 0x7fe6400f5480 2023-11-16T14:07:03.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.569+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6400faf70 con 0x7fe6400f5480 2023-11-16T14:07:03.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.570+0000 7fe63d7fa700 1 -- 172.21.15.112:0/35291505 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe63402b020 con 0x7fe6400f5480 2023-11-16T14:07:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:03.573 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:03.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 msgr2=0x7fe630033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 0x7fe630033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe62c0096a0 tx=0x7fe62c006b40).stop 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 msgr2=0x7fe6401094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fe6340089f0 tx=0x7fe63400b040).stop 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 shutdown_connections 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe630031090 0x7fe630033550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 --2- 172.21.15.112:0/35291505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6400f5480 0x7fe6401094f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 >> 172.21.15.112:0/35291505 conn(0x7fe6400f0be0 msgr2=0x7fe6400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 shutdown_connections 2023-11-16T14:07:03.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:03.572+0000 7fe646c2a700 1 -- 172.21.15.112:0/35291505 wait complete. 2023-11-16T14:07:03.576 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:04.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:02.888785+0000 mgr.smithi061.zcjido (mgr.14156) 172 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:04.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:03.571684+0000 mon.smithi061 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.112:0/35291505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:04.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:04.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:06.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 -- 172.21.15.112:0/3110367267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 msgr2=0x7f642c0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:06.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 --2- 172.21.15.112:0/3110367267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c0f6800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f641c009fb0 tx=0x7f641c009b30).stop 2023-11-16T14:07:06.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 -- 172.21.15.112:0/3110367267 shutdown_connections 2023-11-16T14:07:06.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 --2- 172.21.15.112:0/3110367267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c0f6800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 -- 172.21.15.112:0/3110367267 >> 172.21.15.112:0/3110367267 conn(0x7f642c0f1b40 msgr2=0x7f642c0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 -- 172.21.15.112:0/3110367267 shutdown_connections 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.092+0000 7f6431152700 1 -- 172.21.15.112:0/3110367267 wait complete. 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f6431152700 1 Processor -- start 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f6431152700 1 -- start start 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f6431152700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f6431152700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c106af0 con 0x7f642c0f63e0 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f642bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f642bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58716/0 (socket says 172.21.15.112:58716) 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.093+0000 7f642bfff700 1 -- 172.21.15.112:0/1320190054 learned_addr learned my addr 172.21.15.112:0/1320190054 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f642bfff700 1 -- 172.21.15.112:0/1320190054 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c005040 con 0x7f642c0f63e0 2023-11-16T14:07:06.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f642bfff700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f641c0089f0 tx=0x7f641c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:06.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641c009320 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c106cf0 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f641c013ec0 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641c014d30 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.094+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c107180 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.095+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f641c014420 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.095+0000 7f6428ff9700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 0x7f6414033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.095+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c045bb0 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.095+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f641c0424a0 con 0x7f642c0f63e0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.095+0000 7f642b7fe700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 0x7f6414033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:06.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.096+0000 7f642b7fe700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 0x7f6414033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:06.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.100+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f641c01d7f0 con 0x7f642c0f63e0 2023-11-16T14:07:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:04.889060+0000 mgr.smithi061.zcjido (mgr.14156) 173 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:04.985994+0000 mon.smithi061 (mon.0) 235 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:07:06.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.309+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f642c108000 con 0x7f642c0f63e0 2023-11-16T14:07:06.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.309+0000 7f6428ff9700 1 -- 172.21.15.112:0/1320190054 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f641c039090 con 0x7f642c0f63e0 2023-11-16T14:07:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:06.312 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 msgr2=0x7f6414033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 0x7f6414033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).stop 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 msgr2=0x7f642c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f641c0089f0 tx=0x7f641c00b040).stop 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 shutdown_connections 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6414031090 0x7f6414033550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 --2- 172.21.15.112:0/1320190054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f642c0f63e0 0x7f642c106600 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 >> 172.21.15.112:0/1320190054 conn(0x7f642c0f1b40 msgr2=0x7f642c0f2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 shutdown_connections 2023-11-16T14:07:06.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:06.311+0000 7f6431152700 1 -- 172.21.15.112:0/1320190054 wait complete. 2023-11-16T14:07:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:07.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:06 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:06.310851+0000 mon.smithi061 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.112:0/1320190054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:07.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:07.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:06.889363+0000 mgr.smithi061.zcjido (mgr.14156) 174 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 -- 172.21.15.112:0/1093409839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 msgr2=0x7f36340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 --2- 172.21.15.112:0/1093409839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36340f58a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f3628009fb0 tx=0x7f3628009b30).stop 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 -- 172.21.15.112:0/1093409839 shutdown_connections 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 --2- 172.21.15.112:0/1093409839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36340f58a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 -- 172.21.15.112:0/1093409839 >> 172.21.15.112:0/1093409839 conn(0x7f36340f0be0 msgr2=0x7f36340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 -- 172.21.15.112:0/1093409839 shutdown_connections 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.783+0000 7f3638938700 1 -- 172.21.15.112:0/1093409839 wait complete. 2023-11-16T14:07:08.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f3638938700 1 Processor -- start 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f3638938700 1 -- start start 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f3638938700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f3638938700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3634060320 con 0x7f36340f5480 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f363259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f363259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58738/0 (socket says 172.21.15.112:58738) 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.784+0000 7f363259c700 1 -- 172.21.15.112:0/3053718405 learned_addr learned my addr 172.21.15.112:0/3053718405 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f363259c700 1 -- 172.21.15.112:0/3053718405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3628005040 con 0x7f36340f5480 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f363259c700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f36280089f0 tx=0x7f362800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3628009320 con 0x7f36340f5480 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3634060520 con 0x7f36340f5480 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3628013ec0 con 0x7f36340f5480 2023-11-16T14:07:08.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3628014d30 con 0x7f36340f5480 2023-11-16T14:07:08.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.785+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3634109ba0 con 0x7f36340f5480 2023-11-16T14:07:08.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.786+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f36280095f0 con 0x7f36340f5480 2023-11-16T14:07:08.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.786+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634045bb0 con 0x7f36340f5480 2023-11-16T14:07:08.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.786+0000 7f3622ffd700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 0x7f361c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:08.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.786+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3628042500 con 0x7f36340f5480 2023-11-16T14:07:08.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.786+0000 7f3631d9b700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 0x7f361c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:08.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.787+0000 7f3631d9b700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 0x7f361c0335a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f36240096a0 tx=0x7f3624006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:08.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:08.791+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f362801d7f0 con 0x7f36340f5480 2023-11-16T14:07:09.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.001+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36340faf70 con 0x7f36340f5480 2023-11-16T14:07:09.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.002+0000 7f3622ffd700 1 -- 172.21.15.112:0/3053718405 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f362801d7f0 con 0x7f36340f5480 2023-11-16T14:07:09.005 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:09.006 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:09.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 msgr2=0x7f361c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:09.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 0x7f361c0335a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f36240096a0 tx=0x7f3624006b40).stop 2023-11-16T14:07:09.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 msgr2=0x7f36341094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:09.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f36280089f0 tx=0x7f362800b040).stop 2023-11-16T14:07:09.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 shutdown_connections 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f361c0310e0 0x7f361c0335a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 --2- 172.21.15.112:0/3053718405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36340f5480 0x7f36341094f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 >> 172.21.15.112:0/3053718405 conn(0x7f36340f0be0 msgr2=0x7f36340f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 shutdown_connections 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:09.004+0000 7f3638938700 1 -- 172.21.15.112:0/3053718405 wait complete. 2023-11-16T14:07:09.009 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:08.889662+0000 mgr.smithi061.zcjido (mgr.14156) 175 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:09.003739+0000 mon.smithi061 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.112:0/3053718405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:10.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:10.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 -- 172.21.15.112:0/16921789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 msgr2=0x7f6a300f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 --2- 172.21.15.112:0/16921789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a300f58a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f6a24009fb0 tx=0x7f6a24009b30).stop 2023-11-16T14:07:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 -- 172.21.15.112:0/16921789 shutdown_connections 2023-11-16T14:07:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 --2- 172.21.15.112:0/16921789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a300f58a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 -- 172.21.15.112:0/16921789 >> 172.21.15.112:0/16921789 conn(0x7f6a300f0be0 msgr2=0x7f6a300f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 -- 172.21.15.112:0/16921789 shutdown_connections 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.425+0000 7f6a34bea700 1 -- 172.21.15.112:0/16921789 wait complete. 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.426+0000 7f6a34bea700 1 Processor -- start 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.426+0000 7f6a34bea700 1 -- start start 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.426+0000 7f6a34bea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.426+0000 7f6a34bea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30105be0 con 0x7f6a300f5480 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a2e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a2e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58764/0 (socket says 172.21.15.112:58764) 2023-11-16T14:07:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a2e59c700 1 -- 172.21.15.112:0/982661680 learned_addr learned my addr 172.21.15.112:0/982661680 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:11.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a2e59c700 1 -- 172.21.15.112:0/982661680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a24005040 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a2e59c700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6a240089f0 tx=0x7f6a2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a24009320 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a30105de0 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a30106270 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6a24013ec0 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.427+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a24014d00 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.428+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6a240095f0 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.428+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30045bb0 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.428+0000 7f6a1effd700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 0x7f6a18033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.428+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6a24042490 con 0x7f6a300f5480 2023-11-16T14:07:11.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.428+0000 7f6a2dd9b700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 0x7f6a18033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:11.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.429+0000 7f6a2dd9b700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 0x7f6a18033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6a200096a0 tx=0x7f6a20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:11.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.433+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6a24012b60 con 0x7f6a300f5480 2023-11-16T14:07:11.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.639+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a300faf70 con 0x7f6a300f5480 2023-11-16T14:07:11.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.640+0000 7f6a1effd700 1 -- 172.21.15.112:0/982661680 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a24012330 con 0x7f6a300f5480 2023-11-16T14:07:11.643 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:11.643 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 msgr2=0x7f6a18033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 0x7f6a18033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6a200096a0 tx=0x7f6a20006b40).stop 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 msgr2=0x7f6a301056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6a240089f0 tx=0x7f6a2400b040).stop 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 shutdown_connections 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6a18031090 0x7f6a18033550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 --2- 172.21.15.112:0/982661680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a300f5480 0x7f6a301056f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 >> 172.21.15.112:0/982661680 conn(0x7f6a300f0be0 msgr2=0x7f6a300f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:11.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 shutdown_connections 2023-11-16T14:07:11.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:11.642+0000 7f6a34bea700 1 -- 172.21.15.112:0/982661680 wait complete. 2023-11-16T14:07:11.646 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:10.889914+0000 mgr.smithi061.zcjido (mgr.14156) 176 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:11.641688+0000 mon.smithi061 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.112:0/982661680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:13.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:13.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:12.890260+0000 mgr.smithi061.zcjido (mgr.14156) 177 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:14.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- 172.21.15.112:0/3401096522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 msgr2=0x7f09d80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:14.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/3401096522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80f58d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f09cc009fb0 tx=0x7f09cc009b30).stop 2023-11-16T14:07:14.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- 172.21.15.112:0/3401096522 shutdown_connections 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/3401096522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80f58d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- 172.21.15.112:0/3401096522 >> 172.21.15.112:0/3401096522 conn(0x7f09d80f0c10 msgr2=0x7f09d80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- 172.21.15.112:0/3401096522 shutdown_connections 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- 172.21.15.112:0/3401096522 wait complete. 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 Processor -- start 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.125+0000 7f09dd9d4700 1 -- start start 2023-11-16T14:07:14.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09dd9d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09dd9d4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80f9aa0 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09d6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09d6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:58774/0 (socket says 172.21.15.112:58774) 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09d6ffd700 1 -- 172.21.15.112:0/1288007672 learned_addr learned my addr 172.21.15.112:0/1288007672 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09d6ffd700 1 -- 172.21.15.112:0/1288007672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc005040 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09d6ffd700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f09cc0089f0 tx=0x7f09cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc009320 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d80f9c40 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.126+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f09cc013ec0 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.127+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc014d30 con 0x7f09d80f54b0 2023-11-16T14:07:14.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.127+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d80fa0d0 con 0x7f09d80f54b0 2023-11-16T14:07:14.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.127+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f09cc0263a0 con 0x7f09d80f54b0 2023-11-16T14:07:14.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.127+0000 7f09bf7fe700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 0x7f09c0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:14.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.127+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f09cc042470 con 0x7f09d80f54b0 2023-11-16T14:07:14.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.128+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d80591e0 con 0x7f09d80f54b0 2023-11-16T14:07:14.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.128+0000 7f09d67fc700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 0x7f09c0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:14.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.128+0000 7f09d67fc700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 0x7f09c0033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f09c80096a0 tx=0x7f09c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:14.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.133+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f09cc0266d0 con 0x7f09d80f54b0 2023-11-16T14:07:14.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.341+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09d8045bb0 con 0x7f09d80f54b0 2023-11-16T14:07:14.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.342+0000 7f09bf7fe700 1 -- 172.21.15.112:0/1288007672 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09cc012330 con 0x7f09d80f54b0 2023-11-16T14:07:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:14.346 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 msgr2=0x7f09c0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 0x7f09c0033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f09c80096a0 tx=0x7f09c8006b40).stop 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 msgr2=0x7f09d80fb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f09cc0089f0 tx=0x7f09cc00b040).stop 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 shutdown_connections 2023-11-16T14:07:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f09c0031090 0x7f09c0033550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:14.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 --2- 172.21.15.112:0/1288007672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09d80f54b0 0x7f09d80fb620 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:14.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 >> 172.21.15.112:0/1288007672 conn(0x7f09d80f0c10 msgr2=0x7f09d80f16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:14.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 shutdown_connections 2023-11-16T14:07:14.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:14.344+0000 7f09dd9d4700 1 -- 172.21.15.112:0/1288007672 wait complete. 2023-11-16T14:07:14.349 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:15.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:14.343582+0000 mon.smithi061 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.112:0/1288007672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:15.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:15.710 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:16.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:14.890642+0000 mgr.smithi061.zcjido (mgr.14156) 178 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 -- 172.21.15.112:0/2539341517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 msgr2=0x7faadc0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 --2- 172.21.15.112:0/2539341517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc0f6800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7faacc009fb0 tx=0x7faacc009b30).stop 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 -- 172.21.15.112:0/2539341517 shutdown_connections 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 --2- 172.21.15.112:0/2539341517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc0f6800 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 -- 172.21.15.112:0/2539341517 >> 172.21.15.112:0/2539341517 conn(0x7faadc0f1b40 msgr2=0x7faadc0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 -- 172.21.15.112:0/2539341517 shutdown_connections 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 -- 172.21.15.112:0/2539341517 wait complete. 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.825+0000 7faae1551700 1 Processor -- start 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faae1551700 1 -- start start 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faae1551700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faae1551700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc106b10 con 0x7faadc0f63e0 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faadbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faadbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42396/0 (socket says 172.21.15.112:42396) 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faadbfff700 1 -- 172.21.15.112:0/1775479763 learned_addr learned my addr 172.21.15.112:0/1775479763 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faadbfff700 1 -- 172.21.15.112:0/1775479763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc005040 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.826+0000 7faadbfff700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7faacc0089f0 tx=0x7faacc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc009320 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc106d10 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faacc013ec0 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc014d30 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc1071a0 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.827+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7faacc014420 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.828+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc045bb0 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.828+0000 7faad8ff9700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 0x7faac8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.828+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faacc0423f0 con 0x7faadc0f63e0 2023-11-16T14:07:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.828+0000 7faadb7fe700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 0x7faac8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.828+0000 7faadb7fe700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 0x7faac8033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7faac40096a0 tx=0x7faac4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:16.832+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faacc01d7f0 con 0x7faadc0f63e0 2023-11-16T14:07:17.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.043+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faadc107ff0 con 0x7faadc0f63e0 2023-11-16T14:07:17.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.044+0000 7faad8ff9700 1 -- 172.21.15.112:0/1775479763 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faacc00fe40 con 0x7faadc0f63e0 2023-11-16T14:07:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:17.047 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 msgr2=0x7faac8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 0x7faac8033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7faac40096a0 tx=0x7faac4006b40).stop 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 msgr2=0x7faadc106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7faacc0089f0 tx=0x7faacc00b040).stop 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 shutdown_connections 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7faac8031090 0x7faac8033550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 --2- 172.21.15.112:0/1775479763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faadc0f63e0 0x7faadc106620 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 >> 172.21.15.112:0/1775479763 conn(0x7faadc0f1b40 msgr2=0x7faadc0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 shutdown_connections 2023-11-16T14:07:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:17.046+0000 7faae1551700 1 -- 172.21.15.112:0/1775479763 wait complete. 2023-11-16T14:07:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:16.891076+0000 mgr.smithi061.zcjido (mgr.14156) 179 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:17.045824+0000 mon.smithi061 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.112:0/1775479763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:18.414 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:18.415 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:19.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.531+0000 7ff2828b1700 1 -- 172.21.15.112:0/2269592752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 msgr2=0x7ff27c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:19.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.531+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2269592752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c0f56f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7ff26c009fb0 tx=0x7ff26c009b30).stop 2023-11-16T14:07:19.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.531+0000 7ff2828b1700 1 -- 172.21.15.112:0/2269592752 shutdown_connections 2023-11-16T14:07:19.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.531+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2269592752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c0f56f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.531+0000 7ff2828b1700 1 -- 172.21.15.112:0/2269592752 >> 172.21.15.112:0/2269592752 conn(0x7ff27c0f0a10 msgr2=0x7ff27c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 -- 172.21.15.112:0/2269592752 shutdown_connections 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 -- 172.21.15.112:0/2269592752 wait complete. 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 Processor -- start 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 -- start start 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2828b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c106af0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2818af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2818af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42424/0 (socket says 172.21.15.112:42424) 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.532+0000 7ff2818af700 1 -- 172.21.15.112:0/2871690064 learned_addr learned my addr 172.21.15.112:0/2871690064 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2818af700 1 -- 172.21.15.112:0/2871690064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c005040 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2818af700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ff26c0089f0 tx=0x7ff26c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26c009320 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c106cf0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff26c013ec0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26c014d30 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.533+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c107180 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.534+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff26c014420 con 0x7ff27c0f52d0 2023-11-16T14:07:19.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.534+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c045bb0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.534+0000 7ff2727fc700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 0x7ff268033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:19.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.534+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff26c0424a0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.534+0000 7ff2810ae700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 0x7ff268033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:19.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.535+0000 7ff2810ae700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 0x7ff268033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff2780096a0 tx=0x7ff278006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:19.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.539+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff26c01d7f0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.746+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff27c107fd0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.746+0000 7ff2727fc700 1 -- 172.21.15.112:0/2871690064 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff26c01d7f0 con 0x7ff27c0f52d0 2023-11-16T14:07:19.749 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:19.749 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 msgr2=0x7ff268033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 0x7ff268033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff2780096a0 tx=0x7ff278006b40).stop 2023-11-16T14:07:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 msgr2=0x7ff27c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ff26c0089f0 tx=0x7ff26c00b040).stop 2023-11-16T14:07:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 shutdown_connections 2023-11-16T14:07:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff268031090 0x7ff268033550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 --2- 172.21.15.112:0/2871690064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff27c0f52d0 0x7ff27c106600 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 >> 172.21.15.112:0/2871690064 conn(0x7ff27c0f0a10 msgr2=0x7ff27c05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 shutdown_connections 2023-11-16T14:07:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:19.748+0000 7ff2828b1700 1 -- 172.21.15.112:0/2871690064 wait complete. 2023-11-16T14:07:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:18.891458+0000 mgr.smithi061.zcjido (mgr.14156) 180 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:19.748014+0000 mon.smithi061 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.112:0/2871690064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:21.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:21.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- 172.21.15.112:0/3295275592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 msgr2=0x7ff7400f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 --2- 172.21.15.112:0/3295275592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff7400f56f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ff730009fb0 tx=0x7ff730009b30).stop 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- 172.21.15.112:0/3295275592 shutdown_connections 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 --2- 172.21.15.112:0/3295275592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff7400f56f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- 172.21.15.112:0/3295275592 >> 172.21.15.112:0/3295275592 conn(0x7ff7400f0a30 msgr2=0x7ff7400f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- 172.21.15.112:0/3295275592 shutdown_connections 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- 172.21.15.112:0/3295275592 wait complete. 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 Processor -- start 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.167+0000 7ff74484f700 1 -- start start 2023-11-16T14:07:22.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff74484f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:22.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff74484f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740060300 con 0x7ff7400f52d0 2023-11-16T14:07:22.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff73f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:22.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff73f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42436/0 (socket says 172.21.15.112:42436) 2023-11-16T14:07:22.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff73f7fe700 1 -- 172.21.15.112:0/2941507742 learned_addr learned my addr 172.21.15.112:0/2941507742 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:22.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff73f7fe700 1 -- 172.21.15.112:0/2941507742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730005040 con 0x7ff7400f52d0 2023-11-16T14:07:22.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.168+0000 7ff73f7fe700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7ff7300089f0 tx=0x7ff73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:22.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff730009320 con 0x7ff7400f52d0 2023-11-16T14:07:22.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740060500 con 0x7ff7400f52d0 2023-11-16T14:07:22.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff730013ec0 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff730014d30 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7401099e0 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff730014420 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.169+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7400591e0 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.170+0000 7ff727fff700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 0x7ff728033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.170+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff730042480 con 0x7ff7400f52d0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.170+0000 7ff73effd700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 0x7ff728033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:22.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.170+0000 7ff73effd700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 0x7ff728033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff7340096a0 tx=0x7ff734006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:22.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.175+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff73001d980 con 0x7ff7400f52d0 2023-11-16T14:07:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:20.891793+0000 mgr.smithi061.zcjido (mgr.14156) 181 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:22.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.383+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff740045bb0 con 0x7ff7400f52d0 2023-11-16T14:07:22.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.384+0000 7ff727fff700 1 -- 172.21.15.112:0/2941507742 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff73001d380 con 0x7ff7400f52d0 2023-11-16T14:07:22.441 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:22.441 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 msgr2=0x7ff728033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 0x7ff728033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff7340096a0 tx=0x7ff734006b40).stop 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 msgr2=0x7ff740109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7ff7300089f0 tx=0x7ff73000b040).stop 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 shutdown_connections 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff728031090 0x7ff728033550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 --2- 172.21.15.112:0/2941507742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7400f52d0 0x7ff740109330 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:22.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 >> 172.21.15.112:0/2941507742 conn(0x7ff7400f0a30 msgr2=0x7ff7400f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:22.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 shutdown_connections 2023-11-16T14:07:22.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:22.385+0000 7ff74484f700 1 -- 172.21.15.112:0/2941507742 wait complete. 2023-11-16T14:07:22.443 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:23.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:22.384373+0000 mon.smithi061 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.112:0/2941507742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:23.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:23.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:24.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:22.892089+0000 mgr.smithi061.zcjido (mgr.14156) 182 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:24.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 -- 172.21.15.112:0/832765391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 msgr2=0x7f22000f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:24.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 --2- 172.21.15.112:0/832765391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22000f58a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f21fc009fb0 tx=0x7f21fc009b30).stop 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 -- 172.21.15.112:0/832765391 shutdown_connections 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 --2- 172.21.15.112:0/832765391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22000f58a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 -- 172.21.15.112:0/832765391 >> 172.21.15.112:0/832765391 conn(0x7f22000f0be0 msgr2=0x7f22000f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 -- 172.21.15.112:0/832765391 shutdown_connections 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 -- 172.21.15.112:0/832765391 wait complete. 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.834+0000 7f22087c7700 1 Processor -- start 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f22087c7700 1 -- start start 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f22087c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f22087c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2200106cd0 con 0x7f22000f5480 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f2206563700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:24.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f2206563700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:42458/0 (socket says 172.21.15.112:42458) 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f2206563700 1 -- 172.21.15.112:0/3868479653 learned_addr learned my addr 172.21.15.112:0/3868479653 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f2206563700 1 -- 172.21.15.112:0/3868479653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc005040 con 0x7f22000f5480 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.835+0000 7f2206563700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f21fc0089f0 tx=0x7f21fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21fc009320 con 0x7f22000f5480 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2200106ed0 con 0x7f22000f5480 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f21fc013ec0 con 0x7f22000f5480 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21fc014d30 con 0x7f22000f5480 2023-11-16T14:07:24.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2200107360 con 0x7f22000f5480 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.836+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f21fc0263a0 con 0x7f22000f5480 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.837+0000 7f21f6ffd700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 0x7f21ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.837+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f21fc042380 con 0x7f22000f5480 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.837+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2200045bb0 con 0x7f22000f5480 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.837+0000 7f2205d62700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 0x7f21ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.837+0000 7f2205d62700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 0x7f21ec033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f21f00096a0 tx=0x7f21f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:24.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:24.842+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f21fc012920 con 0x7f22000f5480 2023-11-16T14:07:25.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.050+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2200107fe0 con 0x7f22000f5480 2023-11-16T14:07:25.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.051+0000 7f21f6ffd700 1 -- 172.21.15.112:0/3868479653 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21fc012330 con 0x7f22000f5480 2023-11-16T14:07:25.053 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:25.053 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:25.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 msgr2=0x7f21ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:25.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 0x7f21ec033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f21f00096a0 tx=0x7f21f0006b40).stop 2023-11-16T14:07:25.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 msgr2=0x7f22001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:25.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f21fc0089f0 tx=0x7f21fc00b040).stop 2023-11-16T14:07:25.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 shutdown_connections 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f21ec031090 0x7f21ec033550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 --2- 172.21.15.112:0/3868479653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22000f5480 0x7f22001067e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 >> 172.21.15.112:0/3868479653 conn(0x7f22000f0be0 msgr2=0x7f220005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 shutdown_connections 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:25.052+0000 7f22087c7700 1 -- 172.21.15.112:0/3868479653 wait complete. 2023-11-16T14:07:25.055 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:24.892463+0000 mgr.smithi061.zcjido (mgr.14156) 183 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:25.051447+0000 mon.smithi061 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.112:0/3868479653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:26.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:26.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 -- 172.21.15.112:0/982731685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 msgr2=0x7f5da00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/982731685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da00f58d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5d90009fb0 tx=0x7f5d90009b30).stop 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 -- 172.21.15.112:0/982731685 shutdown_connections 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/982731685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da00f58d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 -- 172.21.15.112:0/982731685 >> 172.21.15.112:0/982731685 conn(0x7f5da00f0c10 msgr2=0x7f5da00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 -- 172.21.15.112:0/982731685 shutdown_connections 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.502+0000 7f5da7fbb700 1 -- 172.21.15.112:0/982731685 wait complete. 2023-11-16T14:07:27.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da7fbb700 1 Processor -- start 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da7fbb700 1 -- start start 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da7fbb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da7fbb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da0105c10 con 0x7f5da00f54b0 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da5d57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da5d57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36808/0 (socket says 172.21.15.112:36808) 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da5d57700 1 -- 172.21.15.112:0/2071209890 learned_addr learned my addr 172.21.15.112:0/2071209890 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.503+0000 7f5da5d57700 1 -- 172.21.15.112:0/2071209890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90005040 con 0x7f5da00f54b0 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5da5d57700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f5d900089f0 tx=0x7f5d9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d90009320 con 0x7f5da00f54b0 2023-11-16T14:07:27.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da0105e10 con 0x7f5da00f54b0 2023-11-16T14:07:27.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5d90013ec0 con 0x7f5da00f54b0 2023-11-16T14:07:27.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d90014d30 con 0x7f5da00f54b0 2023-11-16T14:07:27.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.504+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da01062a0 con 0x7f5da00f54b0 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5d900095f0 con 0x7f5da00f54b0 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da0045bb0 con 0x7f5da00f54b0 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5d967fc700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 0x7f5d8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5d900423e0 con 0x7f5da00f54b0 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5da5556700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 0x7f5d8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:27.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.505+0000 7f5da5556700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 0x7f5d8c033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5d9c0096a0 tx=0x7f5d9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:27.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.510+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5d9001d7f0 con 0x7f5da00f54b0 2023-11-16T14:07:27.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.717+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5da00fafa0 con 0x7f5da00f54b0 2023-11-16T14:07:27.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.718+0000 7f5d967fc700 1 -- 172.21.15.112:0/2071209890 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d9001d7f0 con 0x7f5da00f54b0 2023-11-16T14:07:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:27.720 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:27.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.719+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 msgr2=0x7f5d8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.719+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 0x7f5d8c033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5d9c0096a0 tx=0x7f5d9c006b40).stop 2023-11-16T14:07:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.719+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 msgr2=0x7f5da0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f5d900089f0 tx=0x7f5d9000b040).stop 2023-11-16T14:07:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 shutdown_connections 2023-11-16T14:07:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5d8c031090 0x7f5d8c033550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 --2- 172.21.15.112:0/2071209890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5da00f54b0 0x7f5da0105720 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 >> 172.21.15.112:0/2071209890 conn(0x7f5da00f0c10 msgr2=0x7f5da00f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 shutdown_connections 2023-11-16T14:07:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:27.720+0000 7f5da7fbb700 1 -- 172.21.15.112:0/2071209890 wait complete. 2023-11-16T14:07:27.724 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:26.892844+0000 mgr.smithi061.zcjido (mgr.14156) 184 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:27.718299+0000 mon.smithi061 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.112:0/2071209890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:29.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:33.313 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:33.331 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:28.893240+0000 mgr.smithi061.zcjido (mgr.14156) 185 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:33.331 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:30.893573+0000 mgr.smithi061.zcjido (mgr.14156) 186 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:32.893965+0000 mgr.smithi061.zcjido (mgr.14156) 187 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:34.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- 172.21.15.112:0/166246868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 msgr2=0x7fe8fc0f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 --2- 172.21.15.112:0/166246868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc0f8fb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fe8ec009fb0 tx=0x7fe8ec009b30).stop 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- 172.21.15.112:0/166246868 shutdown_connections 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 --2- 172.21.15.112:0/166246868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc0f8fb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- 172.21.15.112:0/166246868 >> 172.21.15.112:0/166246868 conn(0x7fe8fc0f0a30 msgr2=0x7fe8fc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- 172.21.15.112:0/166246868 shutdown_connections 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- 172.21.15.112:0/166246868 wait complete. 2023-11-16T14:07:34.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 Processor -- start 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.424+0000 7fe903161700 1 -- start start 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe903161700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe903161700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc106aa0 con 0x7fe8fc060360 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe90215f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe90215f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36824/0 (socket says 172.21.15.112:36824) 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe90215f700 1 -- 172.21.15.112:0/3753467520 learned_addr learned my addr 172.21.15.112:0/3753467520 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe90215f700 1 -- 172.21.15.112:0/3753467520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec005040 con 0x7fe8fc060360 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe90215f700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fe8ec009f80 tx=0x7fe8ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:34.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8ec009320 con 0x7fe8fc060360 2023-11-16T14:07:34.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc106ca0 con 0x7fe8fc060360 2023-11-16T14:07:34.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe8ec013680 con 0x7fe8fc060360 2023-11-16T14:07:34.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.425+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8ec01eda0 con 0x7fe8fc060360 2023-11-16T14:07:34.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.426+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc107130 con 0x7fe8fc060360 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.426+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe8ec014420 con 0x7fe8fc060360 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.426+0000 7fe8f2ffd700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.426+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8ec042370 con 0x7fe8fc060360 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.427+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc045bb0 con 0x7fe8fc060360 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.427+0000 7fe90195e700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:34.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.427+0000 7fe90195e700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:34.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.432+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe8ec014700 con 0x7fe8fc060360 2023-11-16T14:07:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.640+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8fc0fbfa0 con 0x7fe8fc060360 2023-11-16T14:07:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.641+0000 7fe8f2ffd700 1 -- 172.21.15.112:0/3753467520 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8ec00faf0 con 0x7fe8fc060360 2023-11-16T14:07:34.643 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:34.643 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 msgr2=0x7fe8e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).stop 2023-11-16T14:07:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 msgr2=0x7fe8fc1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fe8ec009f80 tx=0x7fe8ec00b040).stop 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 shutdown_connections 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe8e8031090 0x7fe8e8033550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 --2- 172.21.15.112:0/3753467520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8fc060360 0x7fe8fc1065b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 >> 172.21.15.112:0/3753467520 conn(0x7fe8fc0f0a30 msgr2=0x7fe8fc0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.642+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 shutdown_connections 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:34.643+0000 7fe903161700 1 -- 172.21.15.112:0/3753467520 wait complete. 2023-11-16T14:07:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:35.007 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:34.641482+0000 mon.smithi061 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.112:0/3753467520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:36.005 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:36.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:34.894365+0000 mgr.smithi061.zcjido (mgr.14156) 188 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:34.983335+0000 mon.smithi061 (mon.0) 246 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:07:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:34.983735+0000 mon.smithi061 (mon.0) 247 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 -- 172.21.15.112:0/2293826012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 msgr2=0x7f64800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 --2- 172.21.15.112:0/2293826012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f64800f69b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6470004760 tx=0x7f6470009ad0).stop 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 -- 172.21.15.112:0/2293826012 shutdown_connections 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 --2- 172.21.15.112:0/2293826012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f64800f69b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 -- 172.21.15.112:0/2293826012 >> 172.21.15.112:0/2293826012 conn(0x7f64800f1cd0 msgr2=0x7f64800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 -- 172.21.15.112:0/2293826012 shutdown_connections 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.113+0000 7f64886b1700 1 -- 172.21.15.112:0/2293826012 wait complete. 2023-11-16T14:07:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f64886b1700 1 Processor -- start 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f64886b1700 1 -- start start 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f64886b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f64886b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480069f00 con 0x7f64800f6590 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f648644d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f648644d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57216/0 (socket says 172.21.15.112:57216) 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f648644d700 1 -- 172.21.15.112:0/1129371879 learned_addr learned my addr 172.21.15.112:0/1129371879 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f648644d700 1 -- 172.21.15.112:0/1129371879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470009d70 con 0x7f64800f6590 2023-11-16T14:07:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f648644d700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6470004170 tx=0x7f6470009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.114+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647000e040 con 0x7f64800f6590 2023-11-16T14:07:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648006a0a0 con 0x7f64800f6590 2023-11-16T14:07:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648006a530 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6470008ef0 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6470014c90 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6470014420 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.115+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64800591e0 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.116+0000 7f6476ffd700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.116+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6470042320 con 0x7f64800f6590 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.116+0000 7f6485c4c700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.116+0000 7f6485c4c700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 0x7f646c033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f647c0096a0 tx=0x7f647c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.120+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6470026300 con 0x7f64800f6590 2023-11-16T14:07:37.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.328+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64800fc080 con 0x7f64800f6590 2023-11-16T14:07:37.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.329+0000 7f6476ffd700 1 -- 172.21.15.112:0/1129371879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6470012b20 con 0x7f64800f6590 2023-11-16T14:07:37.332 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:37.332 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 msgr2=0x7f646c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 0x7f646c033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f647c0096a0 tx=0x7f647c006b40).stop 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 msgr2=0x7f648006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6470004170 tx=0x7f6470009ad0).stop 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 shutdown_connections 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f646c031090 0x7f646c033550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:37.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 --2- 172.21.15.112:0/1129371879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64800f6590 0x7f648006ba80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:37.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 >> 172.21.15.112:0/1129371879 conn(0x7f64800f1cd0 msgr2=0x7f64800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:37.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 shutdown_connections 2023-11-16T14:07:37.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:37.331+0000 7f64886b1700 1 -- 172.21.15.112:0/1129371879 wait complete. 2023-11-16T14:07:37.335 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:36.894755+0000 mgr.smithi061.zcjido (mgr.14156) 189 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:38.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:37.330721+0000 mon.smithi061 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.112:0/1129371879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:38.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:38.708 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 -- 172.21.15.112:0/1777264217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 msgr2=0x7f7ba80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1777264217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80f58a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7b98009fb0 tx=0x7f7b98009b30).stop 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 -- 172.21.15.112:0/1777264217 shutdown_connections 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1777264217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80f58a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 -- 172.21.15.112:0/1777264217 >> 172.21.15.112:0/1777264217 conn(0x7f7ba80f0be0 msgr2=0x7f7ba80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:39.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 -- 172.21.15.112:0/1777264217 shutdown_connections 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.822+0000 7f7bacd17700 1 -- 172.21.15.112:0/1777264217 wait complete. 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7bacd17700 1 Processor -- start 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7bacd17700 1 -- start start 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7bacd17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7bacd17700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba80fb8e0 con 0x7f7ba80f5480 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7ba659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7ba659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57240/0 (socket says 172.21.15.112:57240) 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7ba659c700 1 -- 172.21.15.112:0/1037263402 learned_addr learned my addr 172.21.15.112:0/1037263402 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7ba659c700 1 -- 172.21.15.112:0/1037263402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b98005040 con 0x7f7ba80f5480 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.823+0000 7f7ba659c700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7b980089f0 tx=0x7f7b9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:39.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b98009320 con 0x7f7ba80f5480 2023-11-16T14:07:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba80fba80 con 0x7f7ba80f5480 2023-11-16T14:07:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7b98013ec0 con 0x7f7ba80f5480 2023-11-16T14:07:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b98014d30 con 0x7f7ba80f5480 2023-11-16T14:07:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba80fbf10 con 0x7f7ba80f5480 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.824+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7b98014420 con 0x7f7ba80f5480 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.825+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba80591e0 con 0x7f7ba80f5480 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.825+0000 7f7b96ffd700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 0x7f7b90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.825+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7b980424c0 con 0x7f7ba80f5480 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.825+0000 7f7ba5d9b700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 0x7f7b90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:39.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.825+0000 7f7ba5d9b700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 0x7f7b90033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7b9c0096a0 tx=0x7f7b9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:39.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:39.830+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7b9801d980 con 0x7f7ba80f5480 2023-11-16T14:07:40.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.040+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ba80faf70 con 0x7f7ba80f5480 2023-11-16T14:07:40.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.041+0000 7f7b96ffd700 1 -- 172.21.15.112:0/1037263402 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b9801d380 con 0x7f7ba80f5480 2023-11-16T14:07:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:40.044 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:40.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 msgr2=0x7f7b90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:40.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 0x7f7b90033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7b9c0096a0 tx=0x7f7b9c006b40).stop 2023-11-16T14:07:40.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 msgr2=0x7f7ba80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7b980089f0 tx=0x7f7b9800b040).stop 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 shutdown_connections 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7b90031090 0x7f7b90033550 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 --2- 172.21.15.112:0/1037263402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba80f5480 0x7f7ba80fd460 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 >> 172.21.15.112:0/1037263402 conn(0x7f7ba80f0be0 msgr2=0x7f7ba805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 shutdown_connections 2023-11-16T14:07:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:40.042+0000 7f7bacd17700 1 -- 172.21.15.112:0/1037263402 wait complete. 2023-11-16T14:07:40.047 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:38.895143+0000 mgr.smithi061.zcjido (mgr.14156) 190 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:41.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:40 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:40.042152+0000 mon.smithi061 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.112:0/1037263402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:41.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:41.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:40.895504+0000 mgr.smithi061.zcjido (mgr.14156) 191 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 -- 172.21.15.112:0/1480217371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 msgr2=0x7f719c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 --2- 172.21.15.112:0/1480217371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c0f58a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f7190009fb0 tx=0x7f7190009b30).stop 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 -- 172.21.15.112:0/1480217371 shutdown_connections 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 --2- 172.21.15.112:0/1480217371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c0f58a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 -- 172.21.15.112:0/1480217371 >> 172.21.15.112:0/1480217371 conn(0x7f719c0f0be0 msgr2=0x7f719c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 -- 172.21.15.112:0/1480217371 shutdown_connections 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.532+0000 7f71a3467700 1 -- 172.21.15.112:0/1480217371 wait complete. 2023-11-16T14:07:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a3467700 1 Processor -- start 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a3467700 1 -- start start 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a3467700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a3467700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c060300 con 0x7f719c0f5480 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a1203700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a1203700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57264/0 (socket says 172.21.15.112:57264) 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a1203700 1 -- 172.21.15.112:0/2835903661 learned_addr learned my addr 172.21.15.112:0/2835903661 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a1203700 1 -- 172.21.15.112:0/2835903661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7190005040 con 0x7f719c0f5480 2023-11-16T14:07:42.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.533+0000 7f71a1203700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f71900089f0 tx=0x7f719000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:42.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.534+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7190009320 con 0x7f719c0f5480 2023-11-16T14:07:42.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.534+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c060500 con 0x7f719c0f5480 2023-11-16T14:07:42.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.534+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7190013ec0 con 0x7f719c0f5480 2023-11-16T14:07:42.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.534+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7190014d30 con 0x7f719c0f5480 2023-11-16T14:07:42.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.534+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c109bc0 con 0x7f719c0f5480 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.535+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f71900095f0 con 0x7f719c0f5480 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.535+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c0591e0 con 0x7f719c0f5480 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.535+0000 7f718dffb700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 0x7f7188033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.535+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f71900424e0 con 0x7f719c0f5480 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.535+0000 7f71a0a02700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 0x7f7188033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:42.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.536+0000 7f71a0a02700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 0x7f7188033550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:42.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.540+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f719001d7f0 con 0x7f719c0f5480 2023-11-16T14:07:42.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.749+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f719c045bb0 con 0x7f719c0f5480 2023-11-16T14:07:42.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.750+0000 7f718dffb700 1 -- 172.21.15.112:0/2835903661 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f719000fe40 con 0x7f719c0f5480 2023-11-16T14:07:42.753 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:42.753 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:42.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 msgr2=0x7f7188033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:42.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 0x7f7188033550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).stop 2023-11-16T14:07:42.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 msgr2=0x7f719c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f71900089f0 tx=0x7f719000b040).stop 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 shutdown_connections 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7188031090 0x7f7188033550 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 --2- 172.21.15.112:0/2835903661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f719c0f5480 0x7f719c109510 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 >> 172.21.15.112:0/2835903661 conn(0x7f719c0f0be0 msgr2=0x7f719c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 shutdown_connections 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:42.752+0000 7f71a3467700 1 -- 172.21.15.112:0/2835903661 wait complete. 2023-11-16T14:07:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:42.751342+0000 mon.smithi061 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.112:0/2835903661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:42.895901+0000 mgr.smithi061.zcjido (mgr.14156) 192 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:44.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:44.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:45.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 -- 172.21.15.112:0/3880939009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 msgr2=0x7fd49c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 --2- 172.21.15.112:0/3880939009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c0f58d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd48c009fb0 tx=0x7fd48c009b30).stop 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 -- 172.21.15.112:0/3880939009 shutdown_connections 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 --2- 172.21.15.112:0/3880939009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c0f58d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 -- 172.21.15.112:0/3880939009 >> 172.21.15.112:0/3880939009 conn(0x7fd49c0f0c10 msgr2=0x7fd49c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 -- 172.21.15.112:0/3880939009 shutdown_connections 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.205+0000 7fd4a442c700 1 -- 172.21.15.112:0/3880939009 wait complete. 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a442c700 1 Processor -- start 2023-11-16T14:07:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a442c700 1 -- start start 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a442c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a442c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c106d10 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a21c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a21c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:57280/0 (socket says 172.21.15.112:57280) 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a21c8700 1 -- 172.21.15.112:0/1025464806 learned_addr learned my addr 172.21.15.112:0/1025464806 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.206+0000 7fd4a21c8700 1 -- 172.21.15.112:0/1025464806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c005040 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd4a21c8700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd48c0089f0 tx=0x7fd48c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd48c009320 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c106f10 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd48c013ec0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd48c014d30 con 0x7fd49c0f54b0 2023-11-16T14:07:45.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.207+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c1073a0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.208+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd48c0095f0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.208+0000 7fd492ffd700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 0x7fd488033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:45.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.208+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c045bb0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.208+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd48c0424a0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.208+0000 7fd4a19c7700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 0x7fd488033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:45.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.209+0000 7fd4a19c7700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 0x7fd488033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:45.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.213+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd48c00fe40 con 0x7fd49c0f54b0 2023-11-16T14:07:45.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.421+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd49c1081b0 con 0x7fd49c0f54b0 2023-11-16T14:07:45.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.421+0000 7fd492ffd700 1 -- 172.21.15.112:0/1025464806 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd48c00fe40 con 0x7fd49c0f54b0 2023-11-16T14:07:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:45.425 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 msgr2=0x7fd488033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 0x7fd488033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).stop 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 msgr2=0x7fd49c106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd48c0089f0 tx=0x7fd48c00b040).stop 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 shutdown_connections 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fd488031090 0x7fd488033550 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 --2- 172.21.15.112:0/1025464806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd49c0f54b0 0x7fd49c106820 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 >> 172.21.15.112:0/1025464806 conn(0x7fd49c0f0c10 msgr2=0x7fd49c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 shutdown_connections 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:45.423+0000 7fd4a442c700 1 -- 172.21.15.112:0/1025464806 wait complete. 2023-11-16T14:07:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:46.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:44.896266+0000 mgr.smithi061.zcjido (mgr.14156) 193 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:46.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:45.423122+0000 mon.smithi061 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.112:0/1025464806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:46.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:46.798 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- 172.21.15.112:0/2506205301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 msgr2=0x7fcd6c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 --2- 172.21.15.112:0/2506205301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c0f69b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fcd5c004760 tx=0x7fcd5c009ad0).stop 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- 172.21.15.112:0/2506205301 shutdown_connections 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 --2- 172.21.15.112:0/2506205301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c0f69b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- 172.21.15.112:0/2506205301 >> 172.21.15.112:0/2506205301 conn(0x7fcd6c0f1cd0 msgr2=0x7fcd6c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- 172.21.15.112:0/2506205301 shutdown_connections 2023-11-16T14:07:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- 172.21.15.112:0/2506205301 wait complete. 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 Processor -- start 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.887+0000 7fcd7396b700 1 -- start start 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd7396b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd7396b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c106cb0 con 0x7fcd6c0f6590 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd71707700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd71707700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36056/0 (socket says 172.21.15.112:36056) 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd71707700 1 -- 172.21.15.112:0/940156718 learned_addr learned my addr 172.21.15.112:0/940156718 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd71707700 1 -- 172.21.15.112:0/940156718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c009d70 con 0x7fcd6c0f6590 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.888+0000 7fcd71707700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fcd5c004170 tx=0x7fcd5c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.889+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5c00e040 con 0x7fcd6c0f6590 2023-11-16T14:07:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.889+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c106eb0 con 0x7fcd6c0f6590 2023-11-16T14:07:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.889+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcd5c008ef0 con 0x7fcd6c0f6590 2023-11-16T14:07:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.889+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5c014c90 con 0x7fcd6c0f6590 2023-11-16T14:07:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.889+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c107340 con 0x7fcd6c0f6590 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.890+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcd5c014420 con 0x7fcd6c0f6590 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.890+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c045bb0 con 0x7fcd6c0f6590 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.890+0000 7fcd61ffb700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 0x7fcd58033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.890+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcd5c042300 con 0x7fcd6c0f6590 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.890+0000 7fcd70f06700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 0x7fcd58033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:47.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.891+0000 7fcd70f06700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 0x7fcd58033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd680096a0 tx=0x7fcd68006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:47.895+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcd5c026300 con 0x7fcd6c0f6590 2023-11-16T14:07:48.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.101+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd6c1081f0 con 0x7fcd6c0f6590 2023-11-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.101+0000 7fcd61ffb700 1 -- 172.21.15.112:0/940156718 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd5c012b20 con 0x7fcd6c0f6590 2023-11-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 msgr2=0x7fcd58033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 0x7fcd58033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd680096a0 tx=0x7fcd68006b40).stop 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 msgr2=0x7fcd6c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fcd5c004170 tx=0x7fcd5c009ad0).stop 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 shutdown_connections 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcd58031090 0x7fcd58033550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 --2- 172.21.15.112:0/940156718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd6c0f6590 0x7fcd6c1067c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 >> 172.21.15.112:0/940156718 conn(0x7fcd6c0f1cd0 msgr2=0x7fcd6c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 shutdown_connections 2023-11-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:48.103+0000 7fcd7396b700 1 -- 172.21.15.112:0/940156718 wait complete. 2023-11-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:46.896648+0000 mgr.smithi061.zcjido (mgr.14156) 194 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:49.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:48 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:48.102915+0000 mon.smithi061 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.112:0/940156718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:49.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:49.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:48.896924+0000 mgr.smithi061.zcjido (mgr.14156) 195 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 -- 172.21.15.112:0/4090592837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 msgr2=0x7f30340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 --2- 172.21.15.112:0/4090592837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30340f58a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3024009fb0 tx=0x7f3024009b30).stop 2023-11-16T14:07:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 -- 172.21.15.112:0/4090592837 shutdown_connections 2023-11-16T14:07:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 --2- 172.21.15.112:0/4090592837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30340f58a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 -- 172.21.15.112:0/4090592837 >> 172.21.15.112:0/4090592837 conn(0x7f30340f0be0 msgr2=0x7f30340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 -- 172.21.15.112:0/4090592837 shutdown_connections 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.521+0000 7f303a9b2700 1 -- 172.21.15.112:0/4090592837 wait complete. 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f303a9b2700 1 Processor -- start 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f303a9b2700 1 -- start start 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f303a9b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f303a9b2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3034105be0 con 0x7f30340f5480 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f3033fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f3033fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36074/0 (socket says 172.21.15.112:36074) 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.522+0000 7f3033fff700 1 -- 172.21.15.112:0/3043257427 learned_addr learned my addr 172.21.15.112:0/3043257427 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f3033fff700 1 -- 172.21.15.112:0/3043257427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3024005040 con 0x7f30340f5480 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f3033fff700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f30240089f0 tx=0x7f302400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3024009320 con 0x7f30340f5480 2023-11-16T14:07:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3034105de0 con 0x7f30340f5480 2023-11-16T14:07:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3024013ec0 con 0x7f30340f5480 2023-11-16T14:07:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3024014d30 con 0x7f30340f5480 2023-11-16T14:07:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.523+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3034106270 con 0x7f30340f5480 2023-11-16T14:07:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.524+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f30240095f0 con 0x7f30340f5480 2023-11-16T14:07:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.524+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3034045bb0 con 0x7f30340f5480 2023-11-16T14:07:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.524+0000 7f3030ff9700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 0x7f301c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:50.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.524+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3024042500 con 0x7f30340f5480 2023-11-16T14:07:50.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.524+0000 7f30337fe700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 0x7f301c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:50.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.525+0000 7f30337fe700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 0x7f301c033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:50.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.529+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f302401d7f0 con 0x7f30340f5480 2023-11-16T14:07:50.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.740+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30340faf70 con 0x7f30340f5480 2023-11-16T14:07:50.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.741+0000 7f3030ff9700 1 -- 172.21.15.112:0/3043257427 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f302401d7f0 con 0x7f30340f5480 2023-11-16T14:07:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:50.744 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 msgr2=0x7f301c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 0x7f301c033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).stop 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 msgr2=0x7f30341056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f30240089f0 tx=0x7f302400b040).stop 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 shutdown_connections 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f301c031090 0x7f301c033550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 --2- 172.21.15.112:0/3043257427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30340f5480 0x7f30341056f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:50.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 >> 172.21.15.112:0/3043257427 conn(0x7f30340f0be0 msgr2=0x7f30340f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:50.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 shutdown_connections 2023-11-16T14:07:50.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:50.743+0000 7f303a9b2700 1 -- 172.21.15.112:0/3043257427 wait complete. 2023-11-16T14:07:50.747 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:51.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:50 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:50.742594+0000 mon.smithi061 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.112:0/3043257427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:52.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:52.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:50.897264+0000 mgr.smithi061.zcjido (mgr.14156) 196 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:53.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2918579119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 msgr2=0x7f1f480f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:53.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2918579119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f480f58d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f1f3c009fb0 tx=0x7f1f3c009b30).stop 2023-11-16T14:07:53.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2918579119 shutdown_connections 2023-11-16T14:07:53.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2918579119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f480f58d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:53.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2918579119 >> 172.21.15.112:0/2918579119 conn(0x7f1f480f0c10 msgr2=0x7f1f480f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:53.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2918579119 shutdown_connections 2023-11-16T14:07:53.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.156+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2918579119 wait complete. 2023-11-16T14:07:53.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.157+0000 7f1f4e68e700 1 Processor -- start 2023-11-16T14:07:53.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.157+0000 7f1f4e68e700 1 -- start start 2023-11-16T14:07:53.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.157+0000 7f1f4e68e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:53.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.157+0000 7f1f4e68e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48060360 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.157+0000 7f1f47fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f47fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:36090/0 (socket says 172.21.15.112:36090) 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f47fff700 1 -- 172.21.15.112:0/2736869067 learned_addr learned my addr 172.21.15.112:0/2736869067 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f47fff700 1 -- 172.21.15.112:0/2736869067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c005040 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f47fff700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1f3c0089f0 tx=0x7f1f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3c009320 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f48060560 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1f3c013ec0 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3c014d30 con 0x7f1f480f54b0 2023-11-16T14:07:53.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.158+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f48109b90 con 0x7f1f480f54b0 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.159+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1f3c0263a0 con 0x7f1f480f54b0 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.159+0000 7f1f44ff9700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 0x7f1f30033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.159+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1f3c042460 con 0x7f1f480f54b0 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.159+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f48045bb0 con 0x7f1f480f54b0 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.160+0000 7f1f477fe700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 0x7f1f30033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:53.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.160+0000 7f1f477fe700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 0x7f1f30033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1f380096a0 tx=0x7f1f38006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:53.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.164+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1f3c0266d0 con 0x7f1f480f54b0 2023-11-16T14:07:53.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.369+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f480fb0f0 con 0x7f1f480f54b0 2023-11-16T14:07:53.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.370+0000 7f1f44ff9700 1 -- 172.21.15.112:0/2736869067 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f3c02b020 con 0x7f1f480f54b0 2023-11-16T14:07:53.374 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:53.374 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:53.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 msgr2=0x7f1f30033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:53.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 0x7f1f30033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1f380096a0 tx=0x7f1f38006b40).stop 2023-11-16T14:07:53.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 msgr2=0x7f1f48109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:53.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1f3c0089f0 tx=0x7f1f3c00b040).stop 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 shutdown_connections 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1f30031090 0x7f1f30033550 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 --2- 172.21.15.112:0/2736869067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f480f54b0 0x7f1f48109460 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 >> 172.21.15.112:0/2736869067 conn(0x7f1f480f0c10 msgr2=0x7f1f480f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 shutdown_connections 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:53.371+0000 7f1f4e68e700 1 -- 172.21.15.112:0/2736869067 wait complete. 2023-11-16T14:07:53.376 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:52.897622+0000 mgr.smithi061.zcjido (mgr.14156) 197 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:53.371087+0000 mon.smithi061 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.112:0/2736869067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:54.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:54.722 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 -- 172.21.15.112:0/1433028515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 msgr2=0x7f18700f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 --2- 172.21.15.112:0/1433028515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f18700f69b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1864004760 tx=0x7f1864009ad0).stop 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 -- 172.21.15.112:0/1433028515 shutdown_connections 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 --2- 172.21.15.112:0/1433028515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f18700f69b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 -- 172.21.15.112:0/1433028515 >> 172.21.15.112:0/1433028515 conn(0x7f18700f1cd0 msgr2=0x7f18700f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 -- 172.21.15.112:0/1433028515 shutdown_connections 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 -- 172.21.15.112:0/1433028515 wait complete. 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.824+0000 7f1876f85700 1 Processor -- start 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1876f85700 1 -- start start 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1876f85700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1876f85700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870069f00 con 0x7f18700f6590 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1874d21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1874d21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44780/0 (socket says 172.21.15.112:44780) 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1874d21700 1 -- 172.21.15.112:0/2652554161 learned_addr learned my addr 172.21.15.112:0/2652554161 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1874d21700 1 -- 172.21.15.112:0/2652554161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864009d70 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.825+0000 7f1874d21700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f1864004170 tx=0x7f1864009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.826+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f186400e040 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.826+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187006a0a0 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.826+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1864008ef0 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.826+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1864014c90 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.826+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187006a530 con 0x7f18700f6590 2023-11-16T14:07:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.827+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f18640283f0 con 0x7f18700f6590 2023-11-16T14:07:55.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.827+0000 7f186d7fa700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 0x7f1860033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:55.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.827+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1864042300 con 0x7f18700f6590 2023-11-16T14:07:55.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.827+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18700591e0 con 0x7f18700f6590 2023-11-16T14:07:55.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.827+0000 7f186ffff700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 0x7f1860033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:55.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.828+0000 7f186ffff700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 0x7f1860033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:55.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:55.832+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f18640268f0 con 0x7f18700f6590 2023-11-16T14:07:56.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.042+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1870045bb0 con 0x7f18700f6590 2023-11-16T14:07:56.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.043+0000 7f186d7fa700 1 -- 172.21.15.112:0/2652554161 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1864026300 con 0x7f18700f6590 2023-11-16T14:07:56.174 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:56.174 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 msgr2=0x7f1860033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 0x7f1860033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).stop 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 msgr2=0x7f187006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f1864004170 tx=0x7f1864009ad0).stop 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 shutdown_connections 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1860031090 0x7f1860033550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:56.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 --2- 172.21.15.112:0/2652554161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18700f6590 0x7f187006ba80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:56.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 >> 172.21.15.112:0/2652554161 conn(0x7f18700f1cd0 msgr2=0x7f18700f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:56.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.044+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 shutdown_connections 2023-11-16T14:07:56.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:56.045+0000 7f1876f85700 1 -- 172.21.15.112:0/2652554161 wait complete. 2023-11-16T14:07:56.176 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:54.898085+0000 mgr.smithi061.zcjido (mgr.14156) 198 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:57.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:56 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:56.044369+0000 mon.smithi061 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.112:0/2652554161' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:07:57.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:07:57.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:07:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:56.898412+0000 mgr.smithi061.zcjido (mgr.14156) 199 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:07:58.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 -- 172.21.15.112:0/1446224727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 msgr2=0x7fe6480f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:58.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 --2- 172.21.15.112:0/1446224727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6480f58a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe63c009fb0 tx=0x7fe63c009b30).stop 2023-11-16T14:07:58.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 -- 172.21.15.112:0/1446224727 shutdown_connections 2023-11-16T14:07:58.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 --2- 172.21.15.112:0/1446224727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6480f58a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:58.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 -- 172.21.15.112:0/1446224727 >> 172.21.15.112:0/1446224727 conn(0x7fe6480f0be0 msgr2=0x7fe6480f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:58.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 -- 172.21.15.112:0/1446224727 shutdown_connections 2023-11-16T14:07:58.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.478+0000 7fe64eaab700 1 -- 172.21.15.112:0/1446224727 wait complete. 2023-11-16T14:07:58.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64eaab700 1 Processor -- start 2023-11-16T14:07:58.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64eaab700 1 -- start start 2023-11-16T14:07:58.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64eaab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64eaab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648106cd0 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64c847700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64c847700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44794/0 (socket says 172.21.15.112:44794) 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64c847700 1 -- 172.21.15.112:0/2695533833 learned_addr learned my addr 172.21.15.112:0/2695533833 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.479+0000 7fe64c847700 1 -- 172.21.15.112:0/2695533833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c005040 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe64c847700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe63c0089f0 tx=0x7fe63c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe63c009320 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe648106ed0 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe63c013ec0 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe63c014d30 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.480+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe648107360 con 0x7fe6480f5480 2023-11-16T14:07:58.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe63c0095f0 con 0x7fe6480f5480 2023-11-16T14:07:58.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648045bb0 con 0x7fe6480f5480 2023-11-16T14:07:58.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe6457fa700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 0x7fe638033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:07:58.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe63c0424a0 con 0x7fe6480f5480 2023-11-16T14:07:58.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe647fff700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 0x7fe638033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:07:58.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.481+0000 7fe647fff700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 0x7fe638033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe6340096a0 tx=0x7fe634006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:07:58.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.486+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe63c01d980 con 0x7fe6480f5480 2023-11-16T14:07:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.694+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe648108210 con 0x7fe6480f5480 2023-11-16T14:07:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.695+0000 7fe6457fa700 1 -- 172.21.15.112:0/2695533833 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe63c01d380 con 0x7fe6480f5480 2023-11-16T14:07:58.696 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:07:58.697 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:07:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 msgr2=0x7fe638033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 0x7fe638033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe6340096a0 tx=0x7fe634006b40).stop 2023-11-16T14:07:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 msgr2=0x7fe6481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe63c0089f0 tx=0x7fe63c00b040).stop 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 shutdown_connections 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe638031090 0x7fe638033550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 --2- 172.21.15.112:0/2695533833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6480f5480 0x7fe6481067e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 >> 172.21.15.112:0/2695533833 conn(0x7fe6480f0be0 msgr2=0x7fe64805ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 shutdown_connections 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:07:58.696+0000 7fe64eaab700 1 -- 172.21.15.112:0/2695533833 wait complete. 2023-11-16T14:07:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:07:59.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:58 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:07:58.695320+0000 mon.smithi061 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.112:0/2695533833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:00.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:00.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:07:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:07:58.898660+0000 mgr.smithi061.zcjido (mgr.14156) 200 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.098+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/2304334061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 msgr2=0x7f2cc80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.098+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/2304334061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc80f58a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2cc4009fb0 tx=0x7f2cc4009b30).stop 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/2304334061 shutdown_connections 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/2304334061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc80f58a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/2304334061 >> 172.21.15.112:0/2304334061 conn(0x7f2cc80f0be0 msgr2=0x7f2cc80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/2304334061 shutdown_connections 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/2304334061 wait complete. 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 Processor -- start 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- start start 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccf6d8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc8106ce0 con 0x7f2cc80f5480 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccd474700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccd474700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44804/0 (socket says 172.21.15.112:44804) 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.099+0000 7f2ccd474700 1 -- 172.21.15.112:0/1206997001 learned_addr learned my addr 172.21.15.112:0/1206997001 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2ccd474700 1 -- 172.21.15.112:0/1206997001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4005040 con 0x7f2cc80f5480 2023-11-16T14:08:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2ccd474700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f2cc40089f0 tx=0x7f2cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:01.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc4009320 con 0x7f2cc80f5480 2023-11-16T14:08:01.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc8106ee0 con 0x7f2cc80f5480 2023-11-16T14:08:01.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2cc4013ec0 con 0x7f2cc80f5480 2023-11-16T14:08:01.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc4014d30 con 0x7f2cc80f5480 2023-11-16T14:08:01.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.100+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc8107370 con 0x7f2cc80f5480 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.101+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2cc40095f0 con 0x7f2cc80f5480 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.101+0000 7f2cbdffb700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 0x7f2cb40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.101+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8045bb0 con 0x7f2cc80f5480 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.101+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2cc40424e0 con 0x7f2cc80f5480 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.101+0000 7f2cccc73700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 0x7f2cb40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:01.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.102+0000 7f2cccc73700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 0x7f2cb40335a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2cb80096a0 tx=0x7f2cb8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:01.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.106+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2cc401d7f0 con 0x7f2cc80f5480 2023-11-16T14:08:01.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.313+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cc80fb060 con 0x7f2cc80f5480 2023-11-16T14:08:01.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.314+0000 7f2cbdffb700 1 -- 172.21.15.112:0/1206997001 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cc400fe40 con 0x7f2cc80f5480 2023-11-16T14:08:01.317 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:01.317 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 msgr2=0x7f2cb40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 0x7f2cb40335a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2cb80096a0 tx=0x7f2cb8006b40).stop 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 msgr2=0x7f2cc81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f2cc40089f0 tx=0x7f2cc400b040).stop 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 shutdown_connections 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2cb40310e0 0x7f2cb40335a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 --2- 172.21.15.112:0/1206997001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cc80f5480 0x7f2cc81067f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 >> 172.21.15.112:0/1206997001 conn(0x7f2cc80f0be0 msgr2=0x7f2cc80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 shutdown_connections 2023-11-16T14:08:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:01.316+0000 7f2ccf6d8700 1 -- 172.21.15.112:0/1206997001 wait complete. 2023-11-16T14:08:01.319 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:00.899018+0000 mgr.smithi061.zcjido (mgr.14156) 201 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:01.314729+0000 mon.smithi061 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.112:0/1206997001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:02.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:02.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 -- 172.21.15.112:0/4090603350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 msgr2=0x7f66cc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 --2- 172.21.15.112:0/4090603350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc0f58a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f66bc009fb0 tx=0x7f66bc009b30).stop 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 -- 172.21.15.112:0/4090603350 shutdown_connections 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 --2- 172.21.15.112:0/4090603350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc0f58a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 -- 172.21.15.112:0/4090603350 >> 172.21.15.112:0/4090603350 conn(0x7f66cc0f0be0 msgr2=0x7f66cc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 -- 172.21.15.112:0/4090603350 shutdown_connections 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.864+0000 7f66d387e700 1 -- 172.21.15.112:0/4090603350 wait complete. 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d387e700 1 Processor -- start 2023-11-16T14:08:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d387e700 1 -- start start 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d387e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d387e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc060320 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d161a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d161a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44816/0 (socket says 172.21.15.112:44816) 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d161a700 1 -- 172.21.15.112:0/4256347621 learned_addr learned my addr 172.21.15.112:0/4256347621 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d161a700 1 -- 172.21.15.112:0/4256347621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc005040 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.865+0000 7f66d161a700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f66bc0089f0 tx=0x7f66bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.866+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc009320 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.866+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc060520 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.866+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f66bc013ec0 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.866+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc014d30 con 0x7f66cc0f5480 2023-11-16T14:08:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.866+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc109ba0 con 0x7f66cc0f5480 2023-11-16T14:08:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f66bc014420 con 0x7f66cc0f5480 2023-11-16T14:08:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc045bb0 con 0x7f66cc0f5480 2023-11-16T14:08:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66c1ffb700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 0x7f66b8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f66bc0424c0 con 0x7f66cc0f5480 2023-11-16T14:08:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66d0e19700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 0x7f66b8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.867+0000 7f66d0e19700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 0x7f66b8033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:03.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:03.872+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f66bc01d980 con 0x7f66cc0f5480 2023-11-16T14:08:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:02.899386+0000 mgr.smithi061.zcjido (mgr.14156) 202 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:04.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.082+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66cc0faf70 con 0x7f66cc0f5480 2023-11-16T14:08:04.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.082+0000 7f66c1ffb700 1 -- 172.21.15.112:0/4256347621 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66bc01d380 con 0x7f66cc0f5480 2023-11-16T14:08:04.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:04.085 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:04.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 msgr2=0x7f66b8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:04.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 0x7f66b8033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).stop 2023-11-16T14:08:04.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 msgr2=0x7f66cc1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:04.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f66bc0089f0 tx=0x7f66bc00b040).stop 2023-11-16T14:08:04.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 shutdown_connections 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f66b8031090 0x7f66b8033550 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 --2- 172.21.15.112:0/4256347621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66cc0f5480 0x7f66cc1094f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 >> 172.21.15.112:0/4256347621 conn(0x7f66cc0f0be0 msgr2=0x7f66cc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 shutdown_connections 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:04.084+0000 7f66d387e700 1 -- 172.21.15.112:0/4256347621 wait complete. 2023-11-16T14:08:04.087 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:05.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:04 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:04.083088+0000 mon.smithi061 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.112:0/4256347621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:05.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:05.483 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:06.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:04.899729+0000 mgr.smithi061.zcjido (mgr.14156) 203 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:06.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.502+0000 7f054dcb2700 1 -- 172.21.15.112:0/2050052434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 msgr2=0x7f05480f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:06.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.502+0000 7f054dcb2700 1 --2- 172.21.15.112:0/2050052434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f05480f56f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f053c009fb0 tx=0x7f053c009b30).stop 2023-11-16T14:08:06.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.502+0000 7f054dcb2700 1 -- 172.21.15.112:0/2050052434 shutdown_connections 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.502+0000 7f054dcb2700 1 --2- 172.21.15.112:0/2050052434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f05480f56f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.502+0000 7f054dcb2700 1 -- 172.21.15.112:0/2050052434 >> 172.21.15.112:0/2050052434 conn(0x7f05480f0a30 msgr2=0x7f05480f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 -- 172.21.15.112:0/2050052434 shutdown_connections 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 -- 172.21.15.112:0/2050052434 wait complete. 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 Processor -- start 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 -- start start 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.503+0000 7f054dcb2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0548060300 con 0x7f05480f52d0 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054ccb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054ccb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48408/0 (socket says 172.21.15.112:48408) 2023-11-16T14:08:06.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054ccb0700 1 -- 172.21.15.112:0/3002335578 learned_addr learned my addr 172.21.15.112:0/3002335578 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:06.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054ccb0700 1 -- 172.21.15.112:0/3002335578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c005040 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054ccb0700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f053c0089f0 tx=0x7f053c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c009320 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0548060500 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f053c013ec0 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c014d30 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.504+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481099e0 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.505+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f053c0095f0 con 0x7f05480f52d0 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.505+0000 7f05457fa700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 0x7f0538033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:06.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.505+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05480591e0 con 0x7f05480f52d0 2023-11-16T14:08:06.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.505+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f053c0424a0 con 0x7f05480f52d0 2023-11-16T14:08:06.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.505+0000 7f0547fff700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 0x7f0538033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:06.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.506+0000 7f0547fff700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 0x7f0538033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f05340096a0 tx=0x7f0534006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:06.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.510+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f053c01dd70 con 0x7f05480f52d0 2023-11-16T14:08:06.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.717+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0548045bb0 con 0x7f05480f52d0 2023-11-16T14:08:06.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.718+0000 7f05457fa700 1 -- 172.21.15.112:0/3002335578 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f053c02b020 con 0x7f05480f52d0 2023-11-16T14:08:06.720 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:06.720 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 msgr2=0x7f0538033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 0x7f0538033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f05340096a0 tx=0x7f0534006b40).stop 2023-11-16T14:08:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 msgr2=0x7f0548109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f053c0089f0 tx=0x7f053c00b040).stop 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 shutdown_connections 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0538031090 0x7f0538033550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 --2- 172.21.15.112:0/3002335578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05480f52d0 0x7f0548109330 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 >> 172.21.15.112:0/3002335578 conn(0x7f05480f0a30 msgr2=0x7f05480f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 shutdown_connections 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:06.719+0000 7f054dcb2700 1 -- 172.21.15.112:0/3002335578 wait complete. 2023-11-16T14:08:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:07.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:06 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:06.718345+0000 mon.smithi061 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.112:0/3002335578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:08.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:08.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:06.900158+0000 mgr.smithi061.zcjido (mgr.14156) 204 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:09.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- 172.21.15.112:0/261417015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 msgr2=0x7f4ca00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:09.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/261417015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca00f56f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4c88009fb0 tx=0x7f4c88009b30).stop 2023-11-16T14:08:09.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- 172.21.15.112:0/261417015 shutdown_connections 2023-11-16T14:08:09.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/261417015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca00f56f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- 172.21.15.112:0/261417015 >> 172.21.15.112:0/261417015 conn(0x7f4ca00f0a30 msgr2=0x7f4ca00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- 172.21.15.112:0/261417015 shutdown_connections 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- 172.21.15.112:0/261417015 wait complete. 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 Processor -- start 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.171+0000 7f4c9f59e700 1 -- start start 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:09.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0060320 con 0x7f4ca00f52d0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48430/0 (socket says 172.21.15.112:48430) 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9e59c700 1 -- 172.21.15.112:0/2770867255 learned_addr learned my addr 172.21.15.112:0/2770867255 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9e59c700 1 -- 172.21.15.112:0/2770867255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c88005040 con 0x7f4ca00f52d0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c9e59c700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4c880089f0 tx=0x7f4c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.172+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c88009320 con 0x7f4ca00f52d0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0060520 con 0x7f4ca00f52d0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4c88013ec0 con 0x7f4ca00f52d0 2023-11-16T14:08:09.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c88014d30 con 0x7f4ca00f52d0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca0109a20 con 0x7f4ca00f52d0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4c880095f0 con 0x7f4ca00f52d0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.173+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0045bb0 con 0x7f4ca00f52d0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.174+0000 7f4c96ffd700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 0x7f4c8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.174+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4c880424e0 con 0x7f4ca00f52d0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.174+0000 7f4c9dd9b700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 0x7f4c8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:09.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.174+0000 7f4c9dd9b700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 0x7f4c8c033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:09.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.178+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4c8801d7f0 con 0x7f4ca00f52d0 2023-11-16T14:08:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.389+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca010a230 con 0x7f4ca00f52d0 2023-11-16T14:08:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.390+0000 7f4c96ffd700 1 -- 172.21.15.112:0/2770867255 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c8800fe40 con 0x7f4ca00f52d0 2023-11-16T14:08:09.392 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:09.392 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 msgr2=0x7f4c8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 0x7f4c8c033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).stop 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 msgr2=0x7f4ca0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4c880089f0 tx=0x7f4c8800b040).stop 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 shutdown_connections 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4c8c031090 0x7f4c8c033550 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 --2- 172.21.15.112:0/2770867255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca00f52d0 0x7f4ca0109370 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 >> 172.21.15.112:0/2770867255 conn(0x7f4ca00f0a30 msgr2=0x7f4ca00f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 shutdown_connections 2023-11-16T14:08:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:09.391+0000 7f4c9f59e700 1 -- 172.21.15.112:0/2770867255 wait complete. 2023-11-16T14:08:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:08.900505+0000 mgr.smithi061.zcjido (mgr.14156) 205 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:09.390164+0000 mon.smithi061 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.112:0/2770867255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:10.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:10.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:11.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 -- 172.21.15.112:0/1902657333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 msgr2=0x7f389c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1902657333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c0f58a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f388c009fb0 tx=0x7f388c009b30).stop 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 -- 172.21.15.112:0/1902657333 shutdown_connections 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1902657333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c0f58a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 -- 172.21.15.112:0/1902657333 >> 172.21.15.112:0/1902657333 conn(0x7f389c0f0be0 msgr2=0x7f389c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 -- 172.21.15.112:0/1902657333 shutdown_connections 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.871+0000 7f38a47c7700 1 -- 172.21.15.112:0/1902657333 wait complete. 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a47c7700 1 Processor -- start 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a47c7700 1 -- start start 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a47c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a47c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c105be0 con 0x7f389c0f5480 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a2563700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:11.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a2563700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48442/0 (socket says 172.21.15.112:48442) 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a2563700 1 -- 172.21.15.112:0/1228692240 learned_addr learned my addr 172.21.15.112:0/1228692240 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a2563700 1 -- 172.21.15.112:0/1228692240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c005040 con 0x7f389c0f5480 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.872+0000 7f38a2563700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f388c0089f0 tx=0x7f388c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.873+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f388c009320 con 0x7f389c0f5480 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.873+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c105de0 con 0x7f389c0f5480 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.873+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f388c013ec0 con 0x7f389c0f5480 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.873+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f388c014d30 con 0x7f389c0f5480 2023-11-16T14:08:11.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.873+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c106270 con 0x7f389c0f5480 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.874+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f388c0095f0 con 0x7f389c0f5480 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.874+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c045bb0 con 0x7f389c0f5480 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.874+0000 7f3892ffd700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 0x7f3888033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.874+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f388c042420 con 0x7f389c0f5480 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.874+0000 7f38a1d62700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 0x7f3888033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.875+0000 7f38a1d62700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 0x7f3888033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f38980096a0 tx=0x7f3898006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:11.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:11.879+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f388c01d7f0 con 0x7f389c0f5480 2023-11-16T14:08:12.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.087+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f389c0faf70 con 0x7f389c0f5480 2023-11-16T14:08:12.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.087+0000 7f3892ffd700 1 -- 172.21.15.112:0/1228692240 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f388c00fe40 con 0x7f389c0f5480 2023-11-16T14:08:12.089 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:12.089 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 msgr2=0x7f3888033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 0x7f3888033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f38980096a0 tx=0x7f3898006b40).stop 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 msgr2=0x7f389c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f388c0089f0 tx=0x7f388c00b040).stop 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 shutdown_connections 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3888031090 0x7f3888033550 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 --2- 172.21.15.112:0/1228692240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f389c0f5480 0x7f389c1056f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 >> 172.21.15.112:0/1228692240 conn(0x7f389c0f0be0 msgr2=0x7f389c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 shutdown_connections 2023-11-16T14:08:12.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:12.089+0000 7f38a47c7700 1 -- 172.21.15.112:0/1228692240 wait complete. 2023-11-16T14:08:12.092 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:10.900872+0000 mgr.smithi061.zcjido (mgr.14156) 206 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:13.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:12 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:12.087843+0000 mon.smithi061 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.112:0/1228692240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:13.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:13.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:14.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:12.901311+0000 mgr.smithi061.zcjido (mgr.14156) 207 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.508+0000 7f7c54197700 1 -- 172.21.15.112:0/2202761575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 msgr2=0x7f7c4c0f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.508+0000 7f7c54197700 1 --2- 172.21.15.112:0/2202761575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c0f91c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7c3c009fb0 tx=0x7f7c3c009b30).stop 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.508+0000 7f7c54197700 1 -- 172.21.15.112:0/2202761575 shutdown_connections 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.508+0000 7f7c54197700 1 --2- 172.21.15.112:0/2202761575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c0f91c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.508+0000 7f7c54197700 1 -- 172.21.15.112:0/2202761575 >> 172.21.15.112:0/2202761575 conn(0x7f7c4c0f0be0 msgr2=0x7f7c4c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 -- 172.21.15.112:0/2202761575 shutdown_connections 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 -- 172.21.15.112:0/2202761575 wait complete. 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 Processor -- start 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 -- start start 2023-11-16T14:08:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c54197700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c106c70 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c51f33700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c51f33700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48462/0 (socket says 172.21.15.112:48462) 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.509+0000 7f7c51f33700 1 -- 172.21.15.112:0/660300958 learned_addr learned my addr 172.21.15.112:0/660300958 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c51f33700 1 -- 172.21.15.112:0/660300958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c005040 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c51f33700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f7c3c009f80 tx=0x7f7c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c3c009320 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c106e70 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7c3c013680 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c3c01eda0 con 0x7f7c4c060300 2023-11-16T14:08:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.510+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c107300 con 0x7f7c4c060300 2023-11-16T14:08:14.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.511+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7c3c014420 con 0x7f7c4c060300 2023-11-16T14:08:14.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.511+0000 7f7c427fc700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 0x7f7c38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:14.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.511+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7c3c042410 con 0x7f7c4c060300 2023-11-16T14:08:14.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.511+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c045bb0 con 0x7f7c4c060300 2023-11-16T14:08:14.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.511+0000 7f7c51732700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 0x7f7c38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:14.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.512+0000 7f7c51732700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 0x7f7c38033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7c480096a0 tx=0x7f7c48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.516+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7c3c014700 con 0x7f7c4c060300 2023-11-16T14:08:14.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.726+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c4c107f80 con 0x7f7c4c060300 2023-11-16T14:08:14.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.726+0000 7f7c427fc700 1 -- 172.21.15.112:0/660300958 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c3c01c320 con 0x7f7c4c060300 2023-11-16T14:08:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:14.729 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 msgr2=0x7f7c38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 0x7f7c38033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7c480096a0 tx=0x7f7c48006b40).stop 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 msgr2=0x7f7c4c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f7c3c009f80 tx=0x7f7c3c00b040).stop 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 shutdown_connections 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7c38031090 0x7f7c38033550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 --2- 172.21.15.112:0/660300958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c4c060300 0x7f7c4c106780 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 >> 172.21.15.112:0/660300958 conn(0x7f7c4c0f0be0 msgr2=0x7f7c4c0f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 shutdown_connections 2023-11-16T14:08:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:14.728+0000 7f7c54197700 1 -- 172.21.15.112:0/660300958 wait complete. 2023-11-16T14:08:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:15.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:14.727829+0000 mon.smithi061 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.112:0/660300958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:16.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:16.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:14.901646+0000 mgr.smithi061.zcjido (mgr.14156) 208 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:17.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 -- 172.21.15.112:0/4234215769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 msgr2=0x7efeb80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:17.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 --2- 172.21.15.112:0/4234215769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb80f58a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7efeac009fb0 tx=0x7efeac009b30).stop 2023-11-16T14:08:17.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 -- 172.21.15.112:0/4234215769 shutdown_connections 2023-11-16T14:08:17.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 --2- 172.21.15.112:0/4234215769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb80f58a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 -- 172.21.15.112:0/4234215769 >> 172.21.15.112:0/4234215769 conn(0x7efeb80f0be0 msgr2=0x7efeb80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.211+0000 7efebeda0700 1 -- 172.21.15.112:0/4234215769 shutdown_connections 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.212+0000 7efebeda0700 1 -- 172.21.15.112:0/4234215769 wait complete. 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.212+0000 7efebeda0700 1 Processor -- start 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.212+0000 7efebeda0700 1 -- start start 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.212+0000 7efebeda0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.212+0000 7efebeda0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8060300 con 0x7efeb80f5480 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebcb3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebcb3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45574/0 (socket says 172.21.15.112:45574) 2023-11-16T14:08:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebcb3c700 1 -- 172.21.15.112:0/2291100574 learned_addr learned my addr 172.21.15.112:0/2291100574 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebcb3c700 1 -- 172.21.15.112:0/2291100574 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac005040 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebcb3c700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7efeac0089f0 tx=0x7efeac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeac009320 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb8060500 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efeac013ec0 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeac014d30 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.213+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb8109bc0 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.214+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efeac014420 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.214+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb80591e0 con 0x7efeb80f5480 2023-11-16T14:08:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.214+0000 7efeb57fa700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 0x7efea8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.214+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efeac0424c0 con 0x7efeb80f5480 2023-11-16T14:08:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.214+0000 7efeb7fff700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 0x7efea8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.215+0000 7efeb7fff700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 0x7efea8033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.219+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efeac01d980 con 0x7efeb80f5480 2023-11-16T14:08:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.425+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efeb8045bb0 con 0x7efeb80f5480 2023-11-16T14:08:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.426+0000 7efeb57fa700 1 -- 172.21.15.112:0/2291100574 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efeac01d380 con 0x7efeb80f5480 2023-11-16T14:08:17.429 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:17.429 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 msgr2=0x7efea8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 0x7efea8033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).stop 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 msgr2=0x7efeb8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7efeac0089f0 tx=0x7efeac00b040).stop 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 shutdown_connections 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efea8031090 0x7efea8033550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 --2- 172.21.15.112:0/2291100574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80f5480 0x7efeb8109510 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 >> 172.21.15.112:0/2291100574 conn(0x7efeb80f0be0 msgr2=0x7efeb80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 shutdown_connections 2023-11-16T14:08:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:17.428+0000 7efebeda0700 1 -- 172.21.15.112:0/2291100574 wait complete. 2023-11-16T14:08:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:16.902004+0000 mgr.smithi061.zcjido (mgr.14156) 209 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:17.427831+0000 mon.smithi061 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.112:0/2291100574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:18.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:18.808 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:19.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.958+0000 7f43ded8f700 1 -- 172.21.15.112:0/467014637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 msgr2=0x7f43d80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.958+0000 7f43ded8f700 1 --2- 172.21.15.112:0/467014637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d80f58d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f43cc009fb0 tx=0x7f43cc009b30).stop 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- 172.21.15.112:0/467014637 shutdown_connections 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 --2- 172.21.15.112:0/467014637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d80f58d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- 172.21.15.112:0/467014637 >> 172.21.15.112:0/467014637 conn(0x7f43d80f0c10 msgr2=0x7f43d80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- 172.21.15.112:0/467014637 shutdown_connections 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- 172.21.15.112:0/467014637 wait complete. 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 Processor -- start 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- start start 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.959+0000 7f43ded8f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81072b0 con 0x7f43d80f54b0 2023-11-16T14:08:19.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43dcb2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:19.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43dcb2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45586/0 (socket says 172.21.15.112:45586) 2023-11-16T14:08:19.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43dcb2b700 1 -- 172.21.15.112:0/1512103934 learned_addr learned my addr 172.21.15.112:0/1512103934 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:19.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43dcb2b700 1 -- 172.21.15.112:0/1512103934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc005040 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43dcb2b700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f43cc0089f0 tx=0x7f43cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43cc009320 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d8060300 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f43cc013ec0 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43cc014d30 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.960+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d810bdb0 con 0x7f43d80f54b0 2023-11-16T14:08:19.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.961+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f43cc014420 con 0x7f43d80f54b0 2023-11-16T14:08:19.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.961+0000 7f43d57fa700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 0x7f43c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:19.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.961+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f43cc0423e0 con 0x7f43d80f54b0 2023-11-16T14:08:19.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.961+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d8045bb0 con 0x7f43d80f54b0 2023-11-16T14:08:19.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.962+0000 7f43d7fff700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 0x7f43c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:19.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.962+0000 7f43d7fff700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 0x7f43c8033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f43c40096a0 tx=0x7f43c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:19.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:19.966+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f43cc014750 con 0x7f43d80f54b0 2023-11-16T14:08:20.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.176+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43d80fafa0 con 0x7f43d80f54b0 2023-11-16T14:08:20.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.177+0000 7f43d57fa700 1 -- 172.21.15.112:0/1512103934 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43d80fafa0 con 0x7f43d80f54b0 2023-11-16T14:08:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:20.180 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:20.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.178+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 msgr2=0x7f43c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:20.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.178+0000 7f43ded8f700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 0x7f43c8033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f43c40096a0 tx=0x7f43c4006b40).stop 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.178+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 msgr2=0x7f43d810b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.178+0000 7f43ded8f700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f43cc0089f0 tx=0x7f43cc00b040).stop 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 shutdown_connections 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f43c8031090 0x7f43c8033550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 --2- 172.21.15.112:0/1512103934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43d80f54b0 0x7f43d810b8c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 >> 172.21.15.112:0/1512103934 conn(0x7f43d80f0c10 msgr2=0x7f43d80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 shutdown_connections 2023-11-16T14:08:20.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:20.179+0000 7f43ded8f700 1 -- 172.21.15.112:0/1512103934 wait complete. 2023-11-16T14:08:20.183 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:18.902455+0000 mgr.smithi061.zcjido (mgr.14156) 210 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:21.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:20 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:20.178375+0000 mon.smithi061 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.112:0/1512103934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:21.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:21.547 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:20.902746+0000 mgr.smithi061.zcjido (mgr.14156) 211 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:22.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.619+0000 7f3b64458700 1 -- 172.21.15.112:0/2509427222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 msgr2=0x7f3b5c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:22.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.619+0000 7f3b64458700 1 --2- 172.21.15.112:0/2509427222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c0f69b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3b4c004760 tx=0x7f3b4c009ad0).stop 2023-11-16T14:08:22.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 -- 172.21.15.112:0/2509427222 shutdown_connections 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 --2- 172.21.15.112:0/2509427222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c0f69b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 -- 172.21.15.112:0/2509427222 >> 172.21.15.112:0/2509427222 conn(0x7f3b5c0f1cd0 msgr2=0x7f3b5c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 -- 172.21.15.112:0/2509427222 shutdown_connections 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 -- 172.21.15.112:0/2509427222 wait complete. 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 Processor -- start 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.620+0000 7f3b64458700 1 -- start start 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b64458700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b64458700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c106cb0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b621f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b621f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:45598/0 (socket says 172.21.15.112:45598) 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b621f4700 1 -- 172.21.15.112:0/2872219024 learned_addr learned my addr 172.21.15.112:0/2872219024 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b621f4700 1 -- 172.21.15.112:0/2872219024 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c009d70 con 0x7f3b5c0f6590 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.621+0000 7f3b621f4700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f3b4c013f10 tx=0x7f3b4c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c00e040 con 0x7f3b5c0f6590 2023-11-16T14:08:22.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c106eb0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3b4c007df0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c014d60 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c107340 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.622+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3b4c014520 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.623+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c045bb0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.623+0000 7f3b52ffd700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 0x7f3b48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.623+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3b4c042580 con 0x7f3b5c0f6590 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.623+0000 7f3b619f3700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 0x7f3b48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:22.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.623+0000 7f3b619f3700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 0x7f3b48033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3b580096a0 tx=0x7f3b58006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:22.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.628+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3b4c007960 con 0x7f3b5c0f6590 2023-11-16T14:08:22.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.836+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b5c1081f0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.837+0000 7f3b52ffd700 1 -- 172.21.15.112:0/2872219024 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b4c0262f0 con 0x7f3b5c0f6590 2023-11-16T14:08:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:22.840 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:22.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 msgr2=0x7f3b48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 0x7f3b48033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3b580096a0 tx=0x7f3b58006b40).stop 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 msgr2=0x7f3b5c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f3b4c013f10 tx=0x7f3b4c009ad0).stop 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 shutdown_connections 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3b48031090 0x7f3b48033550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 --2- 172.21.15.112:0/2872219024 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b5c0f6590 0x7f3b5c1067c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 >> 172.21.15.112:0/2872219024 conn(0x7f3b5c0f1cd0 msgr2=0x7f3b5c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 shutdown_connections 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:22.838+0000 7f3b64458700 1 -- 172.21.15.112:0/2872219024 wait complete. 2023-11-16T14:08:22.842 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:22.838342+0000 mon.smithi061 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.112:0/2872219024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:22.903172+0000 mgr.smithi061.zcjido (mgr.14156) 212 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:24.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:24.168 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.317+0000 7fc6f9ffb700 1 -- 172.21.15.112:0/1787100271 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0013860 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1787100271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 msgr2=0x7fc6fc0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1787100271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc0f6800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fc6f0009fb0 tx=0x7fc6f0009b30).stop 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1787100271 shutdown_connections 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1787100271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc0f6800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1787100271 >> 172.21.15.112:0/1787100271 conn(0x7fc6fc0f1b40 msgr2=0x7fc6fc0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1787100271 shutdown_connections 2023-11-16T14:08:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1787100271 wait complete. 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.318+0000 7fc6fbfff700 1 Processor -- start 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6fbfff700 1 -- start start 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6fbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc106b10 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54964/0 (socket says 172.21.15.112:54964) 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6faffd700 1 -- 172.21.15.112:0/1567032494 learned_addr learned my addr 172.21.15.112:0/1567032494 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6faffd700 1 -- 172.21.15.112:0/1567032494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f0005040 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.319+0000 7fc6faffd700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fc6f00089f0 tx=0x7fc6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0009320 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc106d10 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc6f0013ec0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0014d30 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc1071a0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.320+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc6f0014420 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.321+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc045bb0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.321+0000 7fc6e37fe700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 0x7fc6e4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.321+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc6f00424a0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.321+0000 7fc6fa7fc700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 0x7fc6e4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.321+0000 7fc6fa7fc700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 0x7fc6e4033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc6ec0096a0 tx=0x7fc6ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:25.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.326+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc6f001d7f0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.537+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6fc107ff0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.538+0000 7fc6e37fe700 1 -- 172.21.15.112:0/1567032494 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6f001d7f0 con 0x7fc6fc0f63e0 2023-11-16T14:08:25.541 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:25.541 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:25.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.539+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 msgr2=0x7fc6e4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:25.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.539+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 0x7fc6e4033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc6ec0096a0 tx=0x7fc6ec006b40).stop 2023-11-16T14:08:25.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.539+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 msgr2=0x7fc6fc106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.539+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fc6f00089f0 tx=0x7fc6f000b040).stop 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 shutdown_connections 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc6e4031090 0x7fc6e4033550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 --2- 172.21.15.112:0/1567032494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6fc0f63e0 0x7fc6fc106620 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 >> 172.21.15.112:0/1567032494 conn(0x7fc6fc0f1b40 msgr2=0x7fc6fc0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 shutdown_connections 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:25.540+0000 7fc6fbfff700 1 -- 172.21.15.112:0/1567032494 wait complete. 2023-11-16T14:08:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:24.903553+0000 mgr.smithi061.zcjido (mgr.14156) 213 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:25.539458+0000 mon.smithi061 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.112:0/1567032494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:26.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:26.918 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:28.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- 172.21.15.112:0/1528738807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 msgr2=0x7f640c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 --2- 172.21.15.112:0/1528738807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c0f58a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6400009fb0 tx=0x7f6400009b30).stop 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- 172.21.15.112:0/1528738807 shutdown_connections 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 --2- 172.21.15.112:0/1528738807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c0f58a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- 172.21.15.112:0/1528738807 >> 172.21.15.112:0/1528738807 conn(0x7f640c0f0be0 msgr2=0x7f640c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- 172.21.15.112:0/1528738807 shutdown_connections 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- 172.21.15.112:0/1528738807 wait complete. 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 Processor -- start 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.998+0000 7f6412d6b700 1 -- start start 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6412d6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6412d6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c060320 con 0x7f640c0f5480 2023-11-16T14:08:28.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6410b07700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:28.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6410b07700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54980/0 (socket says 172.21.15.112:54980) 2023-11-16T14:08:28.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6410b07700 1 -- 172.21.15.112:0/2156484302 learned_addr learned my addr 172.21.15.112:0/2156484302 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6410b07700 1 -- 172.21.15.112:0/2156484302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400005040 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:27.999+0000 7f6410b07700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f64000089f0 tx=0x7f640000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6400009320 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c060520 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6400013ec0 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6400014d30 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c109ba0 con 0x7f640c0f5480 2023-11-16T14:08:28.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.000+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6400014420 con 0x7f640c0f5480 2023-11-16T14:08:28.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.001+0000 7f64097fa700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 0x7f63fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:28.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.001+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c045bb0 con 0x7f640c0f5480 2023-11-16T14:08:28.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.001+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f64000423e0 con 0x7f640c0f5480 2023-11-16T14:08:28.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.001+0000 7f640bfff700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 0x7f63fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:28.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.001+0000 7f640bfff700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 0x7f63fc033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63f80096a0 tx=0x7f63f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:28.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.006+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f640001d7f0 con 0x7f640c0f5480 2023-11-16T14:08:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.214+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f640c0fb060 con 0x7f640c0f5480 2023-11-16T14:08:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.214+0000 7f64097fa700 1 -- 172.21.15.112:0/2156484302 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f640001d7f0 con 0x7f640c0f5480 2023-11-16T14:08:28.217 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:28.217 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:28.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 msgr2=0x7f63fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:28.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 0x7f63fc033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63f80096a0 tx=0x7f63f8006b40).stop 2023-11-16T14:08:28.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 msgr2=0x7f640c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:28.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f64000089f0 tx=0x7f640000b040).stop 2023-11-16T14:08:28.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 shutdown_connections 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f63fc031090 0x7f63fc033550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 --2- 172.21.15.112:0/2156484302 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640c0f5480 0x7f640c1094f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 >> 172.21.15.112:0/2156484302 conn(0x7f640c0f0be0 msgr2=0x7f640c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 shutdown_connections 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:28.216+0000 7f6412d6b700 1 -- 172.21.15.112:0/2156484302 wait complete. 2023-11-16T14:08:28.220 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:26.903854+0000 mgr.smithi061.zcjido (mgr.14156) 214 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:29.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:28 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:28.215814+0000 mon.smithi061 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.112:0/2156484302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:29.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:29.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:28.904152+0000 mgr.smithi061.zcjido (mgr.14156) 215 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:30.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.615+0000 7f910b686700 1 -- 172.21.15.112:0/1418417046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 msgr2=0x7f91040f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:30.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.615+0000 7f910b686700 1 --2- 172.21.15.112:0/1418417046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91040f58a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f90f8009fb0 tx=0x7f90f8009b30).stop 2023-11-16T14:08:30.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.615+0000 7f910b686700 1 -- 172.21.15.112:0/1418417046 shutdown_connections 2023-11-16T14:08:30.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.615+0000 7f910b686700 1 --2- 172.21.15.112:0/1418417046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91040f58a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.615+0000 7f910b686700 1 -- 172.21.15.112:0/1418417046 >> 172.21.15.112:0/1418417046 conn(0x7f91040f0be0 msgr2=0x7f91040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 -- 172.21.15.112:0/1418417046 shutdown_connections 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 -- 172.21.15.112:0/1418417046 wait complete. 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 Processor -- start 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 -- start start 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f910b686700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9104060360 con 0x7f91040f5480 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f9109422700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f9109422700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55012/0 (socket says 172.21.15.112:55012) 2023-11-16T14:08:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.616+0000 7f9109422700 1 -- 172.21.15.112:0/1012072017 learned_addr learned my addr 172.21.15.112:0/1012072017 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f9109422700 1 -- 172.21.15.112:0/1012072017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f8005040 con 0x7f91040f5480 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f9109422700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f90f80089f0 tx=0x7f90f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90f8009320 con 0x7f91040f5480 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9104060560 con 0x7f91040f5480 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f90f8013ec0 con 0x7f91040f5480 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90f8014d30 con 0x7f91040f5480 2023-11-16T14:08:30.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.617+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9104109be0 con 0x7f91040f5480 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.618+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f90f80095f0 con 0x7f91040f5480 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.618+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9104045bb0 con 0x7f91040f5480 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.618+0000 7f90f5ffb700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 0x7f90f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.618+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f90f80424e0 con 0x7f91040f5480 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.619+0000 7f9108c21700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 0x7f90f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:30.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.619+0000 7f9108c21700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 0x7f90f0033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f91000096a0 tx=0x7f9100006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:30.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.623+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f90f801d7f0 con 0x7f91040f5480 2023-11-16T14:08:30.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.835+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f910410a430 con 0x7f91040f5480 2023-11-16T14:08:30.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.836+0000 7f90f5ffb700 1 -- 172.21.15.112:0/1012072017 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f90f800fe40 con 0x7f91040f5480 2023-11-16T14:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 msgr2=0x7f90f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 0x7f90f0033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f91000096a0 tx=0x7f9100006b40).stop 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 msgr2=0x7f91041094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f90f80089f0 tx=0x7f90f800b040).stop 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 shutdown_connections 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f90f0031090 0x7f90f0033550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 --2- 172.21.15.112:0/1012072017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91040f5480 0x7f91041094b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 >> 172.21.15.112:0/1012072017 conn(0x7f91040f0be0 msgr2=0x7f91040f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 shutdown_connections 2023-11-16T14:08:30.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:30.838+0000 7f910b686700 1 -- 172.21.15.112:0/1012072017 wait complete. 2023-11-16T14:08:30.842 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:31.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:30 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:30.837602+0000 mon.smithi061 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.112:0/1012072017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:32.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:32.205 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:30.904475+0000 mgr.smithi061.zcjido (mgr.14156) 216 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:33.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- 172.21.15.112:0/3487688597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 msgr2=0x7ff3980f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:33.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 --2- 172.21.15.112:0/3487688597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff3980f56f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7ff388009fb0 tx=0x7ff388009b30).stop 2023-11-16T14:08:33.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- 172.21.15.112:0/3487688597 shutdown_connections 2023-11-16T14:08:33.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 --2- 172.21.15.112:0/3487688597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff3980f56f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:33.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- 172.21.15.112:0/3487688597 >> 172.21.15.112:0/3487688597 conn(0x7ff3980f0a30 msgr2=0x7ff3980f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- 172.21.15.112:0/3487688597 shutdown_connections 2023-11-16T14:08:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- 172.21.15.112:0/3487688597 wait complete. 2023-11-16T14:08:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 Processor -- start 2023-11-16T14:08:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.353+0000 7ff39e236700 1 -- start start 2023-11-16T14:08:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39e236700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:33.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39e236700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff398105a00 con 0x7ff3980f52d0 2023-11-16T14:08:33.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39d234700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:33.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39d234700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:55032/0 (socket says 172.21.15.112:55032) 2023-11-16T14:08:33.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39d234700 1 -- 172.21.15.112:0/1017163988 learned_addr learned my addr 172.21.15.112:0/1017163988 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:33.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39d234700 1 -- 172.21.15.112:0/1017163988 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388005040 con 0x7ff3980f52d0 2023-11-16T14:08:33.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff39d234700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7ff3880089f0 tx=0x7ff38800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:33.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.354+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff388009320 con 0x7ff3980f52d0 2023-11-16T14:08:33.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.355+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff398105c00 con 0x7ff3980f52d0 2023-11-16T14:08:33.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.355+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff388013ec0 con 0x7ff3980f52d0 2023-11-16T14:08:33.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.355+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff388014d30 con 0x7ff3980f52d0 2023-11-16T14:08:33.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.355+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff398106090 con 0x7ff3980f52d0 2023-11-16T14:08:33.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff3880095f0 con 0x7ff3980f52d0 2023-11-16T14:08:33.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398045bb0 con 0x7ff3980f52d0 2023-11-16T14:08:33.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff38dffb700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 0x7ff384033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:33.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff3880424a0 con 0x7ff3980f52d0 2023-11-16T14:08:33.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff39ca33700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 0x7ff384033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:33.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.356+0000 7ff39ca33700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 0x7ff384033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff3940096a0 tx=0x7ff394006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:33.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.361+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff38801d7f0 con 0x7ff3980f52d0 2023-11-16T14:08:33.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.568+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff398106f40 con 0x7ff3980f52d0 2023-11-16T14:08:33.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.570+0000 7ff38dffb700 1 -- 172.21.15.112:0/1017163988 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff38801d7f0 con 0x7ff3980f52d0 2023-11-16T14:08:33.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:33.573 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:33.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 msgr2=0x7ff384033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 0x7ff384033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff3940096a0 tx=0x7ff394006b40).stop 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 msgr2=0x7ff398105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7ff3880089f0 tx=0x7ff38800b040).stop 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 shutdown_connections 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff384031090 0x7ff384033550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 --2- 172.21.15.112:0/1017163988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3980f52d0 0x7ff398105510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 >> 172.21.15.112:0/1017163988 conn(0x7ff3980f0a30 msgr2=0x7ff3980f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 shutdown_connections 2023-11-16T14:08:33.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:33.572+0000 7ff39e236700 1 -- 172.21.15.112:0/1017163988 wait complete. 2023-11-16T14:08:33.576 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:32.904844+0000 mgr.smithi061.zcjido (mgr.14156) 217 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:34.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:34.952 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:35.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:34 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:33.571654+0000 mon.smithi061 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.112:0/1017163988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:34.905276+0000 mgr.smithi061.zcjido (mgr.14156) 218 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:34.984406+0000 mon.smithi061 (mon.0) 270 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:08:36.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:34.984712+0000 mon.smithi061 (mon.0) 271 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:08:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 -- 172.21.15.112:0/4043336300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 msgr2=0x7f76080f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 --2- 172.21.15.112:0/4043336300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76080f69b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f75f8004760 tx=0x7f75f8009ad0).stop 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 -- 172.21.15.112:0/4043336300 shutdown_connections 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 --2- 172.21.15.112:0/4043336300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76080f69b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 -- 172.21.15.112:0/4043336300 >> 172.21.15.112:0/4043336300 conn(0x7f76080f1cd0 msgr2=0x7f76080f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 -- 172.21.15.112:0/4043336300 shutdown_connections 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 -- 172.21.15.112:0/4043336300 wait complete. 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.069+0000 7f760fda3700 1 Processor -- start 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760fda3700 1 -- start start 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760fda3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760fda3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608106cb0 con 0x7f76080f6590 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760db3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760db3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53208/0 (socket says 172.21.15.112:53208) 2023-11-16T14:08:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760db3f700 1 -- 172.21.15.112:0/3038982008 learned_addr learned my addr 172.21.15.112:0/3038982008 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760db3f700 1 -- 172.21.15.112:0/3038982008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8009d70 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760db3f700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f75f8004170 tx=0x7f75f8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f800e040 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7608106eb0 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f75f8008ef0 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f8014c90 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.070+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7608107340 con 0x7f76080f6590 2023-11-16T14:08:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.071+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f75f8014420 con 0x7f76080f6590 2023-11-16T14:08:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.071+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7608045bb0 con 0x7f76080f6590 2023-11-16T14:08:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.071+0000 7f75fe7fc700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.071+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f75f80422c0 con 0x7f76080f6590 2023-11-16T14:08:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.072+0000 7f760d33e700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.072+0000 7f760d33e700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 0x7f75f4033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f76040096a0 tx=0x7f7604006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:36.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.076+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f75f8026300 con 0x7f76080f6590 2023-11-16T14:08:36.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.287+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76081081f0 con 0x7f76080f6590 2023-11-16T14:08:36.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.287+0000 7f75fe7fc700 1 -- 172.21.15.112:0/3038982008 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75f8012b20 con 0x7f76080f6590 2023-11-16T14:08:36.289 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:36.289 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 msgr2=0x7f75f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 0x7f75f4033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f76040096a0 tx=0x7f7604006b40).stop 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 msgr2=0x7f76081067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f75f8004170 tx=0x7f75f8009ad0).stop 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 shutdown_connections 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 --2- 172.21.15.112:0/3038982008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76080f6590 0x7f76081067c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 >> 172.21.15.112:0/3038982008 conn(0x7f76080f1cd0 msgr2=0x7f76080f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 shutdown_connections 2023-11-16T14:08:36.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:36.289+0000 7f760fda3700 1 -- 172.21.15.112:0/3038982008 wait complete. 2023-11-16T14:08:36.292 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:37.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:36 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:36.287797+0000 mon.smithi061 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.112:0/3038982008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:37.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:37.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:38.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:36.905728+0000 mgr.smithi061.zcjido (mgr.14156) 219 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/1822771080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 msgr2=0x7fdf380f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/1822771080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf380f69b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fdf28009fb0 tx=0x7fdf28009b30).stop 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/1822771080 shutdown_connections 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/1822771080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf380f69b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/1822771080 >> 172.21.15.112:0/1822771080 conn(0x7fdf380f1cf0 msgr2=0x7fdf380f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/1822771080 shutdown_connections 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.724+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/1822771080 wait complete. 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf3e5cd700 1 Processor -- start 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf3e5cd700 1 -- start start 2023-11-16T14:08:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf3e5cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf3e5cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38106cc0 con 0x7fdf380f6590 2023-11-16T14:08:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53220/0 (socket says 172.21.15.112:53220) 2023-11-16T14:08:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf37fff700 1 -- 172.21.15.112:0/4036260256 learned_addr learned my addr 172.21.15.112:0/4036260256 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.725+0000 7fdf37fff700 1 -- 172.21.15.112:0/4036260256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28005040 con 0x7fdf380f6590 2023-11-16T14:08:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf37fff700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fdf280089f0 tx=0x7fdf2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28009320 con 0x7fdf380f6590 2023-11-16T14:08:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf38106ec0 con 0x7fdf380f6590 2023-11-16T14:08:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf38107350 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf28004490 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.726+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28014c50 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.727+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdf28014420 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.727+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c004fa0 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.727+0000 7fdf34ff9700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 0x7fdf24033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.727+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdf280423f0 con 0x7fdf380f6590 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.727+0000 7fdf377fe700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 0x7fdf24033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.728+0000 7fdf377fe700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 0x7fdf24033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fdf200096a0 tx=0x7fdf20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.732+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf280095f0 con 0x7fdf380f6590 2023-11-16T14:08:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.944+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf1c005670 con 0x7fdf380f6590 2023-11-16T14:08:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.945+0000 7fdf34ff9700 1 -- 172.21.15.112:0/4036260256 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf28012750 con 0x7fdf380f6590 2023-11-16T14:08:38.947 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:38.947 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:38.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 msgr2=0x7fdf24033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:38.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 0x7fdf24033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fdf200096a0 tx=0x7fdf20006b40).stop 2023-11-16T14:08:38.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 msgr2=0x7fdf381067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:38.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fdf280089f0 tx=0x7fdf2800b040).stop 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 shutdown_connections 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf24031090 0x7fdf24033550 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.946+0000 7fdf3e5cd700 1 --2- 172.21.15.112:0/4036260256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf380f6590 0x7fdf381067d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.947+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 >> 172.21.15.112:0/4036260256 conn(0x7fdf380f1cf0 msgr2=0x7fdf380f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.947+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 shutdown_connections 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:38.947+0000 7fdf3e5cd700 1 -- 172.21.15.112:0/4036260256 wait complete. 2023-11-16T14:08:38.949 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:39.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:38 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:38.945436+0000 mon.smithi061 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.112:0/4036260256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:38.906074+0000 mgr.smithi061.zcjido (mgr.14156) 220 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:40.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:40.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:41.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 -- 172.21.15.112:0/1619936127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 msgr2=0x7f16700f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:41.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 --2- 172.21.15.112:0/1619936127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16700f69b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1664004760 tx=0x7f1664009ad0).stop 2023-11-16T14:08:41.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 -- 172.21.15.112:0/1619936127 shutdown_connections 2023-11-16T14:08:41.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 --2- 172.21.15.112:0/1619936127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16700f69b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:41.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 -- 172.21.15.112:0/1619936127 >> 172.21.15.112:0/1619936127 conn(0x7f16700f1cd0 msgr2=0x7f16700f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:41.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 -- 172.21.15.112:0/1619936127 shutdown_connections 2023-11-16T14:08:41.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 -- 172.21.15.112:0/1619936127 wait complete. 2023-11-16T14:08:41.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.393+0000 7f1676889700 1 Processor -- start 2023-11-16T14:08:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f1676889700 1 -- start start 2023-11-16T14:08:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f1676889700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f1676889700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1670106cb0 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f166ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f166ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53250/0 (socket says 172.21.15.112:53250) 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f166ffff700 1 -- 172.21.15.112:0/3529555458 learned_addr learned my addr 172.21.15.112:0/3529555458 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f166ffff700 1 -- 172.21.15.112:0/3529555458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664009d70 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.394+0000 7f166ffff700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f1664004170 tx=0x7f1664009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166400e040 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1664008ef0 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1670106eb0 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1664014c90 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1670107340 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.395+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1664014420 con 0x7f16700f6590 2023-11-16T14:08:41.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.396+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1670045bb0 con 0x7f16700f6590 2023-11-16T14:08:41.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.396+0000 7f166cff9700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 0x7f1658033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:41.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.396+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1664042240 con 0x7f16700f6590 2023-11-16T14:08:41.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.396+0000 7f166f7fe700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 0x7f1658033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:41.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.396+0000 7f166f7fe700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 0x7f1658033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f16600096a0 tx=0x7f1660006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:41.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.401+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1664012d00 con 0x7f16700f6590 2023-11-16T14:08:41.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.608+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16701081f0 con 0x7f16700f6590 2023-11-16T14:08:41.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.608+0000 7f166cff9700 1 -- 172.21.15.112:0/3529555458 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16640126b0 con 0x7f16700f6590 2023-11-16T14:08:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:41.610 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 msgr2=0x7f1658033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 0x7f1658033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f16600096a0 tx=0x7f1660006b40).stop 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 msgr2=0x7f16701067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f1664004170 tx=0x7f1664009ad0).stop 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 shutdown_connections 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f1658031090 0x7f1658033550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 --2- 172.21.15.112:0/3529555458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700f6590 0x7f16701067c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 >> 172.21.15.112:0/3529555458 conn(0x7f16700f1cd0 msgr2=0x7f16700f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:41.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 shutdown_connections 2023-11-16T14:08:41.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:41.610+0000 7f1676889700 1 -- 172.21.15.112:0/3529555458 wait complete. 2023-11-16T14:08:41.613 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:40.906373+0000 mgr.smithi061.zcjido (mgr.14156) 221 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:41.608963+0000 mon.smithi061 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.112:0/3529555458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:42.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:42.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:42.906645+0000 mgr.smithi061.zcjido (mgr.14156) 222 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:44.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- 172.21.15.112:0/2696214265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 msgr2=0x7f59840f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 --2- 172.21.15.112:0/2696214265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f59840f6800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5978009fb0 tx=0x7f5978009b30).stop 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- 172.21.15.112:0/2696214265 shutdown_connections 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 --2- 172.21.15.112:0/2696214265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f59840f6800 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- 172.21.15.112:0/2696214265 >> 172.21.15.112:0/2696214265 conn(0x7f59840f1b40 msgr2=0x7f59840f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- 172.21.15.112:0/2696214265 shutdown_connections 2023-11-16T14:08:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- 172.21.15.112:0/2696214265 wait complete. 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 Processor -- start 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.127+0000 7f5989f0b700 1 -- start start 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5989f0b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5989f0b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984106b10 con 0x7f59840f63e0 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5988f09700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5988f09700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53272/0 (socket says 172.21.15.112:53272) 2023-11-16T14:08:44.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5988f09700 1 -- 172.21.15.112:0/1838496150 learned_addr learned my addr 172.21.15.112:0/1838496150 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5988f09700 1 -- 172.21.15.112:0/1838496150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978005040 con 0x7f59840f63e0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5988f09700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f59780089f0 tx=0x7f597800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5978009320 con 0x7f59840f63e0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5984106d10 con 0x7f59840f63e0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5978013ec0 con 0x7f59840f63e0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.128+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5978014d30 con 0x7f59840f63e0 2023-11-16T14:08:44.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.129+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841071a0 con 0x7f59840f63e0 2023-11-16T14:08:44.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.129+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f59780095f0 con 0x7f59840f63e0 2023-11-16T14:08:44.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.129+0000 7f59817fa700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 0x7f5974033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:44.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.129+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f59780424a0 con 0x7f59840f63e0 2023-11-16T14:08:44.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.129+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984045bb0 con 0x7f59840f63e0 2023-11-16T14:08:44.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.130+0000 7f5983fff700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 0x7f5974033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:44.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.130+0000 7f5983fff700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 0x7f5974033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f59700096a0 tx=0x7f5970006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:44.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.135+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f597801d7f0 con 0x7f59840f63e0 2023-11-16T14:08:44.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.343+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59840fbfc0 con 0x7f59840f63e0 2023-11-16T14:08:44.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.344+0000 7f59817fa700 1 -- 172.21.15.112:0/1838496150 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f597801d7f0 con 0x7f59840f63e0 2023-11-16T14:08:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:44.346 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:44.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 msgr2=0x7f5974033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:44.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 0x7f5974033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f59700096a0 tx=0x7f5970006b40).stop 2023-11-16T14:08:44.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 msgr2=0x7f5984106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:44.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f59780089f0 tx=0x7f597800b040).stop 2023-11-16T14:08:44.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 shutdown_connections 2023-11-16T14:08:44.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f5974031090 0x7f5974033550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:44.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 --2- 172.21.15.112:0/1838496150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59840f63e0 0x7f5984106620 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:44.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 >> 172.21.15.112:0/1838496150 conn(0x7f59840f1b40 msgr2=0x7f59840f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:44.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 shutdown_connections 2023-11-16T14:08:44.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:44.345+0000 7f5989f0b700 1 -- 172.21.15.112:0/1838496150 wait complete. 2023-11-16T14:08:44.349 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:45.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:44 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:44.344361+0000 mon.smithi061 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.112:0/1838496150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:45.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:45.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:46.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:44.906964+0000 mgr.smithi061.zcjido (mgr.14156) 223 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 -- 172.21.15.112:0/1759566453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 msgr2=0x7f01a00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 --2- 172.21.15.112:0/1759566453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a00f69b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0194004760 tx=0x7f0194009ad0).stop 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 -- 172.21.15.112:0/1759566453 shutdown_connections 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 --2- 172.21.15.112:0/1759566453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a00f69b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 -- 172.21.15.112:0/1759566453 >> 172.21.15.112:0/1759566453 conn(0x7f01a00f1cd0 msgr2=0x7f01a00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 -- 172.21.15.112:0/1759566453 shutdown_connections 2023-11-16T14:08:46.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 -- 172.21.15.112:0/1759566453 wait complete. 2023-11-16T14:08:46.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.822+0000 7f01a6431700 1 Processor -- start 2023-11-16T14:08:46.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f01a6431700 1 -- start start 2023-11-16T14:08:46.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f01a6431700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:46.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f01a6431700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0069f00 con 0x7f01a00f6590 2023-11-16T14:08:46.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f019ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:46.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f019ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46872/0 (socket says 172.21.15.112:46872) 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.823+0000 7f019ffff700 1 -- 172.21.15.112:0/807069083 learned_addr learned my addr 172.21.15.112:0/807069083 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f019ffff700 1 -- 172.21.15.112:0/807069083 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0194009d70 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f019ffff700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f0194004170 tx=0x7f0194009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f019400e040 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a006a0a0 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0194008ef0 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a006a530 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.824+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0194014c90 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.825+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0194014420 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.825+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a00591e0 con 0x7f01a00f6590 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.825+0000 7f019cff9700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 0x7f0188033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:46.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.825+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0194042320 con 0x7f01a00f6590 2023-11-16T14:08:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.825+0000 7f019f7fe700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 0x7f0188033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.826+0000 7f019f7fe700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 0x7f0188033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f01900096a0 tx=0x7f0190006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:46.830+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0194026300 con 0x7f01a00f6590 2023-11-16T14:08:47.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.040+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01a00fc080 con 0x7f01a00f6590 2023-11-16T14:08:47.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.040+0000 7f019cff9700 1 -- 172.21.15.112:0/807069083 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0194012b20 con 0x7f01a00f6590 2023-11-16T14:08:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:47.043 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 msgr2=0x7f0188033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 0x7f0188033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f01900096a0 tx=0x7f0190006b40).stop 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 msgr2=0x7f01a006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f0194004170 tx=0x7f0194009ad0).stop 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 shutdown_connections 2023-11-16T14:08:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f0188031090 0x7f0188033550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 --2- 172.21.15.112:0/807069083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01a00f6590 0x7f01a006ba80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 >> 172.21.15.112:0/807069083 conn(0x7f01a00f1cd0 msgr2=0x7f01a00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 shutdown_connections 2023-11-16T14:08:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:47.042+0000 7f01a6431700 1 -- 172.21.15.112:0/807069083 wait complete. 2023-11-16T14:08:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:46.907311+0000 mgr.smithi061.zcjido (mgr.14156) 224 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:48.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:47.041126+0000 mon.smithi061 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.112:0/807069083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:48.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:48.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 -- 172.21.15.112:0/3290688400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 msgr2=0x7f45800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 --2- 172.21.15.112:0/3290688400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f45800f69b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4574004760 tx=0x7f4574009ad0).stop 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 -- 172.21.15.112:0/3290688400 shutdown_connections 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 --2- 172.21.15.112:0/3290688400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f45800f69b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 -- 172.21.15.112:0/3290688400 >> 172.21.15.112:0/3290688400 conn(0x7f45800f1cd0 msgr2=0x7f45800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 -- 172.21.15.112:0/3290688400 shutdown_connections 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 -- 172.21.15.112:0/3290688400 wait complete. 2023-11-16T14:08:49.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.563+0000 7f4585fbe700 1 Processor -- start 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f4585fbe700 1 -- start start 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f4585fbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f4585fbe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580069f00 con 0x7f45800f6590 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f457f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f457f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46900/0 (socket says 172.21.15.112:46900) 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f457f7fe700 1 -- 172.21.15.112:0/2987962680 learned_addr learned my addr 172.21.15.112:0/2987962680 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f457f7fe700 1 -- 172.21.15.112:0/2987962680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574009d70 con 0x7f45800f6590 2023-11-16T14:08:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.564+0000 7f457f7fe700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4574004170 tx=0x7f4574009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457400e040 con 0x7f45800f6590 2023-11-16T14:08:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458006a0a0 con 0x7f45800f6590 2023-11-16T14:08:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4574008ef0 con 0x7f45800f6590 2023-11-16T14:08:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4574014c90 con 0x7f45800f6590 2023-11-16T14:08:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458006a530 con 0x7f45800f6590 2023-11-16T14:08:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4574014420 con 0x7f45800f6590 2023-11-16T14:08:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.565+0000 7f4567fff700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 0x7f4568033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.566+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4574042300 con 0x7f45800f6590 2023-11-16T14:08:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.566+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45800591e0 con 0x7f45800f6590 2023-11-16T14:08:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.566+0000 7f457effd700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 0x7f4568033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.566+0000 7f457effd700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 0x7f4568033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f45700096a0 tx=0x7f4570006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.571+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4574012d00 con 0x7f45800f6590 2023-11-16T14:08:49.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.783+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45800fc080 con 0x7f45800f6590 2023-11-16T14:08:49.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.784+0000 7f4567fff700 1 -- 172.21.15.112:0/2987962680 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4574012b20 con 0x7f45800f6590 2023-11-16T14:08:49.786 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:49.786 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 msgr2=0x7f4568033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 0x7f4568033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f45700096a0 tx=0x7f4570006b40).stop 2023-11-16T14:08:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 msgr2=0x7f458006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4574004170 tx=0x7f4574009ad0).stop 2023-11-16T14:08:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 shutdown_connections 2023-11-16T14:08:49.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4568031090 0x7f4568033550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:49.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 --2- 172.21.15.112:0/2987962680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45800f6590 0x7f458006ba80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:49.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 >> 172.21.15.112:0/2987962680 conn(0x7f45800f1cd0 msgr2=0x7f45800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:49.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 shutdown_connections 2023-11-16T14:08:49.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:49.785+0000 7f4585fbe700 1 -- 172.21.15.112:0/2987962680 wait complete. 2023-11-16T14:08:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:48.907703+0000 mgr.smithi061.zcjido (mgr.14156) 225 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:49.784498+0000 mon.smithi061 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.112:0/2987962680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:51.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:51.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 -- 172.21.15.112:0/1323319717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 msgr2=0x7fa4d80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/1323319717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d80f69b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fa4cc004760 tx=0x7fa4cc009ad0).stop 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 -- 172.21.15.112:0/1323319717 shutdown_connections 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/1323319717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d80f69b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 -- 172.21.15.112:0/1323319717 >> 172.21.15.112:0/1323319717 conn(0x7fa4d80f1cd0 msgr2=0x7fa4d80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 -- 172.21.15.112:0/1323319717 shutdown_connections 2023-11-16T14:08:52.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.253+0000 7fa4deb1c700 1 -- 172.21.15.112:0/1323319717 wait complete. 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4deb1c700 1 Processor -- start 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4deb1c700 1 -- start start 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4deb1c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4deb1c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8106cb0 con 0x7fa4d80f6590 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4dc8b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4dc8b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46910/0 (socket says 172.21.15.112:46910) 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4dc8b8700 1 -- 172.21.15.112:0/3333647702 learned_addr learned my addr 172.21.15.112:0/3333647702 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.254+0000 7fa4dc8b8700 1 -- 172.21.15.112:0/3333647702 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc009d70 con 0x7fa4d80f6590 2023-11-16T14:08:52.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4dc8b8700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa4cc004170 tx=0x7fa4cc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:52.259 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:50.908079+0000 mgr.smithi061.zcjido (mgr.14156) 226 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc00e040 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d8106eb0 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa4cc008ef0 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc014c90 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.255+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d8107340 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa4cc014420 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d8045bb0 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4d57fa700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 0x7fa4c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa4cc0421f0 con 0x7fa4d80f6590 2023-11-16T14:08:52.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4d7fff700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 0x7fa4c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:52.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.256+0000 7fa4d7fff700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 0x7fa4c8033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa4c40096a0 tx=0x7fa4c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:52.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.261+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa4cc026300 con 0x7fa4d80f6590 2023-11-16T14:08:52.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.469+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4d81081f0 con 0x7fa4d80f6590 2023-11-16T14:08:52.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.470+0000 7fa4d57fa700 1 -- 172.21.15.112:0/3333647702 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4cc012b20 con 0x7fa4d80f6590 2023-11-16T14:08:52.473 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:52.473 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:52.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 msgr2=0x7fa4c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 0x7fa4c8033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa4c40096a0 tx=0x7fa4c4006b40).stop 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 msgr2=0x7fa4d81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa4cc004170 tx=0x7fa4cc009ad0).stop 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 shutdown_connections 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa4c8031090 0x7fa4c8033550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 --2- 172.21.15.112:0/3333647702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80f6590 0x7fa4d81067c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 >> 172.21.15.112:0/3333647702 conn(0x7fa4d80f1cd0 msgr2=0x7fa4d80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 shutdown_connections 2023-11-16T14:08:52.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:52.472+0000 7fa4deb1c700 1 -- 172.21.15.112:0/3333647702 wait complete. 2023-11-16T14:08:52.476 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:53.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:52 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:52.471762+0000 mon.smithi061 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.112:0/3333647702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:53.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:53.829 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:54.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:52.908362+0000 mgr.smithi061.zcjido (mgr.14156) 227 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- 172.21.15.112:0/2293142616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 msgr2=0x7f7dc00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 --2- 172.21.15.112:0/2293142616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc00f56f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7dbc009fb0 tx=0x7f7dbc009b30).stop 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- 172.21.15.112:0/2293142616 shutdown_connections 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 --2- 172.21.15.112:0/2293142616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc00f56f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- 172.21.15.112:0/2293142616 >> 172.21.15.112:0/2293142616 conn(0x7f7dc00f0a30 msgr2=0x7f7dc00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- 172.21.15.112:0/2293142616 shutdown_connections 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- 172.21.15.112:0/2293142616 wait complete. 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 Processor -- start 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.913+0000 7f7dc753f700 1 -- start start 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc753f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc753f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc0105a00 con 0x7f7dc00f52d0 2023-11-16T14:08:54.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc653d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc653d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46938/0 (socket says 172.21.15.112:46938) 2023-11-16T14:08:54.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc653d700 1 -- 172.21.15.112:0/3534303117 learned_addr learned my addr 172.21.15.112:0/3534303117 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc653d700 1 -- 172.21.15.112:0/3534303117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc005040 con 0x7f7dc00f52d0 2023-11-16T14:08:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc653d700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f7dbc0089f0 tx=0x7f7dbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dbc009320 con 0x7f7dc00f52d0 2023-11-16T14:08:54.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc0105c00 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7dbc013ec0 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.914+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dbc014d30 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.915+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc0106090 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.915+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7dbc0095f0 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.915+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc0045bb0 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.915+0000 7f7db6ffd700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 0x7f7dac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.915+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7dbc0424a0 con 0x7f7dc00f52d0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.916+0000 7f7dc5d3c700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 0x7f7dac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:54.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.916+0000 7f7dc5d3c700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 0x7f7dac033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7db00096a0 tx=0x7f7db0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:54.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:54.920+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7dbc01d7f0 con 0x7f7dc00f52d0 2023-11-16T14:08:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.128+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dc0106f40 con 0x7f7dc00f52d0 2023-11-16T14:08:55.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.129+0000 7f7db6ffd700 1 -- 172.21.15.112:0/3534303117 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7dbc01d7f0 con 0x7f7dc00f52d0 2023-11-16T14:08:55.132 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:55.132 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 msgr2=0x7f7dac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 0x7f7dac033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7db00096a0 tx=0x7f7db0006b40).stop 2023-11-16T14:08:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 msgr2=0x7f7dc0105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f7dbc0089f0 tx=0x7f7dbc00b040).stop 2023-11-16T14:08:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 shutdown_connections 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7dac031090 0x7f7dac033550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 --2- 172.21.15.112:0/3534303117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc00f52d0 0x7f7dc0105510 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 >> 172.21.15.112:0/3534303117 conn(0x7f7dc00f0a30 msgr2=0x7f7dc00f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 shutdown_connections 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:55.131+0000 7f7dc753f700 1 -- 172.21.15.112:0/3534303117 wait complete. 2023-11-16T14:08:55.135 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:54.908774+0000 mgr.smithi061.zcjido (mgr.14156) 228 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:56.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:55.130658+0000 mon.smithi061 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.112:0/3534303117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:56.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:56.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 -- 172.21.15.112:0/2375062283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 msgr2=0x7f16700fb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 --2- 172.21.15.112:0/2375062283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16700fb430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f1658009fb0 tx=0x7f1658009b30).stop 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 -- 172.21.15.112:0/2375062283 shutdown_connections 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 --2- 172.21.15.112:0/2375062283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16700fb430 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 -- 172.21.15.112:0/2375062283 >> 172.21.15.112:0/2375062283 conn(0x7f16700f0a30 msgr2=0x7f16700f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 -- 172.21.15.112:0/2375062283 shutdown_connections 2023-11-16T14:08:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.595+0000 7f1674c95700 1 -- 172.21.15.112:0/2375062283 wait complete. 2023-11-16T14:08:57.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f1674c95700 1 Processor -- start 2023-11-16T14:08:57.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f1674c95700 1 -- start start 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f1674c95700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f1674c95700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1670108cb0 con 0x7f16700602e0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f166f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f166f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34214/0 (socket says 172.21.15.112:34214) 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f166f7fe700 1 -- 172.21.15.112:0/98471276 learned_addr learned my addr 172.21.15.112:0/98471276 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.596+0000 7f166f7fe700 1 -- 172.21.15.112:0/98471276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1658005040 con 0x7f16700602e0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f166f7fe700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f1658009f80 tx=0x7f165800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1658009320 con 0x7f16700602e0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1670108eb0 con 0x7f16700602e0 2023-11-16T14:08:57.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1658013680 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f165801eda0 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.597+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1670109340 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f165801e420 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f1667fff700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 0x7f165c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1658042440 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1670045bb0 con 0x7f16700602e0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f166effd700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 0x7f165c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:08:57.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.598+0000 7f166effd700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 0x7f165c033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16600096a0 tx=0x7f1660006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:08:57.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.603+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f165801e700 con 0x7f16700602e0 2023-11-16T14:08:57.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.815+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f167010a090 con 0x7f16700602e0 2023-11-16T14:08:57.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.816+0000 7f1667fff700 1 -- 172.21.15.112:0/98471276 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f165801cb10 con 0x7f16700602e0 2023-11-16T14:08:57.819 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:08:57.819 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:08:57.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.817+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 msgr2=0x7f165c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.817+0000 7f1674c95700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 0x7f165c033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16600096a0 tx=0x7f1660006b40).stop 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.817+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 msgr2=0x7f16701087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.817+0000 7f1674c95700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f1658009f80 tx=0x7f165800b040).stop 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 shutdown_connections 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f165c031090 0x7f165c033550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 --2- 172.21.15.112:0/98471276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16700602e0 0x7f16701087c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 >> 172.21.15.112:0/98471276 conn(0x7f16700f0a30 msgr2=0x7f16700f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 shutdown_connections 2023-11-16T14:08:57.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:08:57.818+0000 7f1674c95700 1 -- 172.21.15.112:0/98471276 wait complete. 2023-11-16T14:08:57.822 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:08:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:56.909164+0000 mgr.smithi061.zcjido (mgr.14156) 229 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:08:58.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:08:57.817481+0000 mon.smithi061 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.112:0/98471276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:08:59.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:08:59.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 -- 172.21.15.112:0/1242747930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 msgr2=0x7fc5440f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 --2- 172.21.15.112:0/1242747930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc5440f69b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fc538004760 tx=0x7fc538009ad0).stop 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 -- 172.21.15.112:0/1242747930 shutdown_connections 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 --2- 172.21.15.112:0/1242747930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc5440f69b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 -- 172.21.15.112:0/1242747930 >> 172.21.15.112:0/1242747930 conn(0x7fc5440f1cd0 msgr2=0x7fc5440f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 -- 172.21.15.112:0/1242747930 shutdown_connections 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.244+0000 7fc54b15a700 1 -- 172.21.15.112:0/1242747930 wait complete. 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.245+0000 7fc54b15a700 1 Processor -- start 2023-11-16T14:09:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.245+0000 7fc54b15a700 1 -- start start 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.245+0000 7fc54b15a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.245+0000 7fc54b15a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc544069f00 con 0x7fc5440f6590 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc548ef6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc548ef6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34224/0 (socket says 172.21.15.112:34224) 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc548ef6700 1 -- 172.21.15.112:0/3695348560 learned_addr learned my addr 172.21.15.112:0/3695348560 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc548ef6700 1 -- 172.21.15.112:0/3695348560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538009d70 con 0x7fc5440f6590 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc548ef6700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fc538013f10 tx=0x7fc538009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc53800e040 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54406a0a0 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc538007df0 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc538014d60 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.246+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54406a530 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.247+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc538014520 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.247+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5440591e0 con 0x7fc5440f6590 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.247+0000 7fc5417fa700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 0x7fc5340335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.247+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc538042580 con 0x7fc5440f6590 2023-11-16T14:09:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.247+0000 7fc543fff700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 0x7fc5340335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.248+0000 7fc543fff700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 0x7fc5340335a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc5300096a0 tx=0x7fc530006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:00.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.252+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc538028910 con 0x7fc5440f6590 2023-11-16T14:09:00.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:08:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:08:58.909595+0000 mgr.smithi061.zcjido (mgr.14156) 230 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:00.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.462+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5440fc080 con 0x7fc5440f6590 2023-11-16T14:09:00.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.463+0000 7fc5417fa700 1 -- 172.21.15.112:0/3695348560 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5380262f0 con 0x7fc5440f6590 2023-11-16T14:09:00.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:00.466 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 msgr2=0x7fc5340335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 0x7fc5340335a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc5300096a0 tx=0x7fc530006b40).stop 2023-11-16T14:09:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 msgr2=0x7fc54406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fc538013f10 tx=0x7fc538009ad0).stop 2023-11-16T14:09:00.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 shutdown_connections 2023-11-16T14:09:00.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fc5340310e0 0x7fc5340335a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:00.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 --2- 172.21.15.112:0/3695348560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5440f6590 0x7fc54406ba80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:00.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 >> 172.21.15.112:0/3695348560 conn(0x7fc5440f1cd0 msgr2=0x7fc5440f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:00.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 shutdown_connections 2023-11-16T14:09:00.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:00.465+0000 7fc54b15a700 1 -- 172.21.15.112:0/3695348560 wait complete. 2023-11-16T14:09:00.471 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:01.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:00 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:00.464371+0000 mon.smithi061 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.112:0/3695348560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:01.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:01.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:02.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:00.909930+0000 mgr.smithi061.zcjido (mgr.14156) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:02.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- 172.21.15.112:0/4055904418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 msgr2=0x7fdf140f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:02.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/4055904418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf140f5700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fdf04009fb0 tx=0x7fdf04009b30).stop 2023-11-16T14:09:02.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- 172.21.15.112:0/4055904418 shutdown_connections 2023-11-16T14:09:02.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/4055904418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf140f5700 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- 172.21.15.112:0/4055904418 >> 172.21.15.112:0/4055904418 conn(0x7fdf140f0a40 msgr2=0x7fdf140f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- 172.21.15.112:0/4055904418 shutdown_connections 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- 172.21.15.112:0/4055904418 wait complete. 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 Processor -- start 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- start start 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.978+0000 7fdf1b14a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141059f0 con 0x7fdf140f52e0 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1a148700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1a148700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34228/0 (socket says 172.21.15.112:34228) 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1a148700 1 -- 172.21.15.112:0/3291507108 learned_addr learned my addr 172.21.15.112:0/3291507108 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1a148700 1 -- 172.21.15.112:0/3291507108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04005040 con 0x7fdf140f52e0 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1a148700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fdf040089f0 tx=0x7fdf0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:02.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04009320 con 0x7fdf140f52e0 2023-11-16T14:09:02.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf14105bf0 con 0x7fdf140f52e0 2023-11-16T14:09:02.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf04013ec0 con 0x7fdf140f52e0 2023-11-16T14:09:02.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04014d30 con 0x7fdf140f52e0 2023-11-16T14:09:02.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.979+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf14106080 con 0x7fdf140f52e0 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.980+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdf04014420 con 0x7fdf140f52e0 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.980+0000 7fdf0affd700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 0x7fdf00033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.980+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf14045bb0 con 0x7fdf140f52e0 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.980+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdf040424c0 con 0x7fdf140f52e0 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.980+0000 7fdf19947700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 0x7fdf00033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:02.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.981+0000 7fdf19947700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 0x7fdf00033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf100096a0 tx=0x7fdf10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:02.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:02.985+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf0401d980 con 0x7fdf140f52e0 2023-11-16T14:09:03.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.193+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf140fadd0 con 0x7fdf140f52e0 2023-11-16T14:09:03.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.194+0000 7fdf0affd700 1 -- 172.21.15.112:0/3291507108 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf0401d380 con 0x7fdf140f52e0 2023-11-16T14:09:03.197 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:03.197 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 msgr2=0x7fdf00033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 0x7fdf00033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf100096a0 tx=0x7fdf10006b40).stop 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 msgr2=0x7fdf14105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fdf040089f0 tx=0x7fdf0400b040).stop 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 shutdown_connections 2023-11-16T14:09:03.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdf00031090 0x7fdf00033550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:03.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 --2- 172.21.15.112:0/3291507108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf140f52e0 0x7fdf14105500 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:03.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 >> 172.21.15.112:0/3291507108 conn(0x7fdf140f0a40 msgr2=0x7fdf140f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:03.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 shutdown_connections 2023-11-16T14:09:03.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:03.195+0000 7fdf1b14a700 1 -- 172.21.15.112:0/3291507108 wait complete. 2023-11-16T14:09:03.199 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:09:02.547351+0000 mgr.smithi061.zcjido (mgr.14156) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon node-exporter.smithi061 ... 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Verifying port 9100 ... 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 1 from systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr Job for ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service failed because a timeout was exceeded. 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr See "systemctl status ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service" and "journalctl -xe" for details. 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:09:04.009 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ports=daemon_ports) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-16T14:09:04.010 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: Failed command: systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: yield (conn, connr) 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: code, '\n'.join(err))) 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon node-exporter.smithi061 ... 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Verifying port 9100 ... 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 1 from systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr Job for ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service failed because a timeout was exceeded. 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr See "systemctl status ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service" and "journalctl -xe" for details. 2023-11-16T14:09:04.011 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ports=daemon_ports) 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-11-16T14:09:04.012 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: Failed command: systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:09:02.547653+0000 mgr.smithi061.zcjido (mgr.14156) 233 : cephadm [ERR] Failed while placing node-exporter.smithi061 on smithi061: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: /bin/podman: stderr Error: no such container ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-node-exporter.smithi061 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Deploy daemon node-exporter.smithi061 ... 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Verifying port 9100 ... 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Non-zero exit code 1 from systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr Job for ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service failed because a timeout was exceeded. 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: systemctl: stderr See "systemctl status ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061.service" and "journalctl -xe" for details. 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: Traceback (most recent call last): 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-11-16T14:09:04.013 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: main() 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: r = ctx.func(ctx) 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: return func(ctx) 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: ports=daemon_ports) 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: File "/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-11-16T14:09:04.014 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-16T14:09:04.015 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: RuntimeError: Failed command: systemctl start ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@node-exporter.smithi061 2023-11-16T14:09:04.015 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cephadm 2023-11-16T14:09:02.554287+0000 mgr.smithi061.zcjido (mgr.14156) 234 : cephadm [INF] Deploying daemon prometheus.smithi061 on smithi061 2023-11-16T14:09:04.015 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:02.910203+0000 mgr.smithi061.zcjido (mgr.14156) 235 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:04.015 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:03.195254+0000 mon.smithi061 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.112:0/3291507108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:04.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:04.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 -- 172.21.15.112:0/2264588769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 msgr2=0x7f33bc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 --2- 172.21.15.112:0/2264588769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc0f58a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f33ac009fb0 tx=0x7f33ac009b30).stop 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 -- 172.21.15.112:0/2264588769 shutdown_connections 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 --2- 172.21.15.112:0/2264588769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc0f58a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 -- 172.21.15.112:0/2264588769 >> 172.21.15.112:0/2264588769 conn(0x7f33bc0f0be0 msgr2=0x7f33bc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 -- 172.21.15.112:0/2264588769 shutdown_connections 2023-11-16T14:09:05.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.659+0000 7f33c39bc700 1 -- 172.21.15.112:0/2264588769 wait complete. 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c39bc700 1 Processor -- start 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c39bc700 1 -- start start 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c39bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c39bc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc060320 con 0x7f33bc0f5480 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c1758700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c1758700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48716/0 (socket says 172.21.15.112:48716) 2023-11-16T14:09:05.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.660+0000 7f33c1758700 1 -- 172.21.15.112:0/1945334018 learned_addr learned my addr 172.21.15.112:0/1945334018 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33c1758700 1 -- 172.21.15.112:0/1945334018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac005040 con 0x7f33bc0f5480 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33c1758700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f33ac0089f0 tx=0x7f33ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ac009320 con 0x7f33bc0f5480 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc060520 con 0x7f33bc0f5480 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f33ac013ec0 con 0x7f33bc0f5480 2023-11-16T14:09:05.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ac014d30 con 0x7f33bc0f5480 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.661+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc109ba0 con 0x7f33bc0f5480 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.662+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f33ac0263a0 con 0x7f33bc0f5480 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.662+0000 7f33b1ffb700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 0x7f33a80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.662+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f33ac042460 con 0x7f33bc0f5480 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.662+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc045bb0 con 0x7f33bc0f5480 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.662+0000 7f33c0f57700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 0x7f33a80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:05.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.663+0000 7f33c0f57700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 0x7f33a80335a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:05.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.667+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f33ac0266d0 con 0x7f33bc0f5480 2023-11-16T14:09:05.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.878+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33bc0fb060 con 0x7f33bc0f5480 2023-11-16T14:09:05.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.878+0000 7f33b1ffb700 1 -- 172.21.15.112:0/1945334018 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f33ac02b020 con 0x7f33bc0f5480 2023-11-16T14:09:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:05.882 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:05.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 msgr2=0x7f33a80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:05.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 0x7f33a80335a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).stop 2023-11-16T14:09:05.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 msgr2=0x7f33bc1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:05.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f33ac0089f0 tx=0x7f33ac00b040).stop 2023-11-16T14:09:05.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 shutdown_connections 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f33a80310e0 0x7f33a80335a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 --2- 172.21.15.112:0/1945334018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33bc0f5480 0x7f33bc1094f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 >> 172.21.15.112:0/1945334018 conn(0x7f33bc0f0be0 msgr2=0x7f33bc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 shutdown_connections 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:05.880+0000 7f33c39bc700 1 -- 172.21.15.112:0/1945334018 wait complete. 2023-11-16T14:09:05.884 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:04.910716+0000 mgr.smithi061.zcjido (mgr.14156) 236 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:04.992495+0000 mon.smithi061 (mon.0) 283 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' 2023-11-16T14:09:06.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:05.879943+0000 mon.smithi061 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.112:0/1945334018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:07.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:07.310 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:08.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:07 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:06.911109+0000 mgr.smithi061.zcjido (mgr.14156) 237 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:08.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 -- 172.21.15.112:0/1584971585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 msgr2=0x7ff8700f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:08.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 --2- 172.21.15.112:0/1584971585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8700f58a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff860009fb0 tx=0x7ff860009b30).stop 2023-11-16T14:09:08.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 -- 172.21.15.112:0/1584971585 shutdown_connections 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 --2- 172.21.15.112:0/1584971585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8700f58a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 -- 172.21.15.112:0/1584971585 >> 172.21.15.112:0/1584971585 conn(0x7ff8700f0be0 msgr2=0x7ff8700f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 -- 172.21.15.112:0/1584971585 shutdown_connections 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.402+0000 7ff875114700 1 -- 172.21.15.112:0/1584971585 wait complete. 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff875114700 1 Processor -- start 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff875114700 1 -- start start 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff875114700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff875114700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870106ce0 con 0x7ff8700f5480 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff86ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff86ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48730/0 (socket says 172.21.15.112:48730) 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff86ed9d700 1 -- 172.21.15.112:0/1760545034 learned_addr learned my addr 172.21.15.112:0/1760545034 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff86ed9d700 1 -- 172.21.15.112:0/1760545034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860005040 con 0x7ff8700f5480 2023-11-16T14:09:08.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.403+0000 7ff86ed9d700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff8600089f0 tx=0x7ff86000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:08.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff860009320 con 0x7ff8700f5480 2023-11-16T14:09:08.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff870106ee0 con 0x7ff8700f5480 2023-11-16T14:09:08.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff860013ec0 con 0x7ff8700f5480 2023-11-16T14:09:08.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff860014d30 con 0x7ff8700f5480 2023-11-16T14:09:08.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff870107370 con 0x7ff8700f5480 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff8600095f0 con 0x7ff8700f5480 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.404+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff870045bb0 con 0x7ff8700f5480 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.405+0000 7ff8577fe700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.405+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff860042420 con 0x7ff8700f5480 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.405+0000 7ff86e59c700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:08.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.405+0000 7ff86e59c700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 0x7ff858033550 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff8640096a0 tx=0x7ff864006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:08.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.409+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff86001d7f0 con 0x7ff8700f5480 2023-11-16T14:09:08.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.619+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8700faf70 con 0x7ff8700f5480 2023-11-16T14:09:08.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.620+0000 7ff8577fe700 1 -- 172.21.15.112:0/1760545034 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff86000fe40 con 0x7ff8700f5480 2023-11-16T14:09:08.623 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:08.623 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:08.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 msgr2=0x7ff858033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:08.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 0x7ff858033550 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff8640096a0 tx=0x7ff864006b40).stop 2023-11-16T14:09:08.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 msgr2=0x7ff8701067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff8600089f0 tx=0x7ff86000b040).stop 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 shutdown_connections 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff858031090 0x7ff858033550 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 --2- 172.21.15.112:0/1760545034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8700f5480 0x7ff8701067f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 >> 172.21.15.112:0/1760545034 conn(0x7ff8700f0be0 msgr2=0x7ff8700f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 shutdown_connections 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:08.621+0000 7ff875114700 1 -- 172.21.15.112:0/1760545034 wait complete. 2023-11-16T14:09:08.625 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:09.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:08.621201+0000 mon.smithi061 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.112:0/1760545034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:09.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:09.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:10.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:09 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:08.911377+0000 mgr.smithi061.zcjido (mgr.14156) 238 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 -- 172.21.15.112:0/3244245730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 msgr2=0x7efdc00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 --2- 172.21.15.112:0/3244245730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc00f58a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7efdb0009fb0 tx=0x7efdb0009b30).stop 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 -- 172.21.15.112:0/3244245730 shutdown_connections 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 --2- 172.21.15.112:0/3244245730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc00f58a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 -- 172.21.15.112:0/3244245730 >> 172.21.15.112:0/3244245730 conn(0x7efdc00f0be0 msgr2=0x7efdc00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 -- 172.21.15.112:0/3244245730 shutdown_connections 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.067+0000 7efdc843b700 1 -- 172.21.15.112:0/3244245730 wait complete. 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.068+0000 7efdc843b700 1 Processor -- start 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.068+0000 7efdc843b700 1 -- start start 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.068+0000 7efdc843b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.068+0000 7efdc843b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0108ed0 con 0x7efdc00f5480 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc61d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc61d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48742/0 (socket says 172.21.15.112:48742) 2023-11-16T14:09:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc61d7700 1 -- 172.21.15.112:0/3308107414 learned_addr learned my addr 172.21.15.112:0/3308107414 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc61d7700 1 -- 172.21.15.112:0/3308107414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0005040 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc61d7700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7efdb00089f0 tx=0x7efdb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdb0009320 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc01090d0 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efdb0013ec0 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdb0014d30 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.069+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc0109560 con 0x7efdc00f5480 2023-11-16T14:09:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.070+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efdb00263a0 con 0x7efdc00f5480 2023-11-16T14:09:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.070+0000 7efdb6ffd700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 0x7efdac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.070+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efdb00423b0 con 0x7efdc00f5480 2023-11-16T14:09:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.070+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc00591e0 con 0x7efdc00f5480 2023-11-16T14:09:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.070+0000 7efdc59d6700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 0x7efdac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.071+0000 7efdc59d6700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 0x7efdac033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efdbc0096a0 tx=0x7efdbc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.075+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efdb00266d0 con 0x7efdc00f5480 2023-11-16T14:09:11.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.284+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdc0045bb0 con 0x7efdc00f5480 2023-11-16T14:09:11.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.285+0000 7efdb6ffd700 1 -- 172.21.15.112:0/3308107414 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdb002b020 con 0x7efdc00f5480 2023-11-16T14:09:11.288 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:11.288 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:11.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 msgr2=0x7efdac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:11.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 0x7efdac033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efdbc0096a0 tx=0x7efdbc006b40).stop 2023-11-16T14:09:11.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 msgr2=0x7efdc01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7efdb00089f0 tx=0x7efdb000b040).stop 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 shutdown_connections 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7efdac031090 0x7efdac033550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 --2- 172.21.15.112:0/3308107414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdc00f5480 0x7efdc01089e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 >> 172.21.15.112:0/3308107414 conn(0x7efdc00f0be0 msgr2=0x7efdc005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 shutdown_connections 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:11.287+0000 7efdc843b700 1 -- 172.21.15.112:0/3308107414 wait complete. 2023-11-16T14:09:11.290 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:10.911638+0000 mgr.smithi061.zcjido (mgr.14156) 239 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:12.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:11.286665+0000 mon.smithi061 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.112:0/3308107414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:12.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:12.659 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- 172.21.15.112:0/553894775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 msgr2=0x7f31d40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 --2- 172.21.15.112:0/553894775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d40f56f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f31c8009fb0 tx=0x7f31c8009b30).stop 2023-11-16T14:09:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- 172.21.15.112:0/553894775 shutdown_connections 2023-11-16T14:09:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 --2- 172.21.15.112:0/553894775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d40f56f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- 172.21.15.112:0/553894775 >> 172.21.15.112:0/553894775 conn(0x7f31d40f0a30 msgr2=0x7f31d40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- 172.21.15.112:0/553894775 shutdown_connections 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- 172.21.15.112:0/553894775 wait complete. 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 Processor -- start 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.820+0000 7f31d9430700 1 -- start start 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d9430700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d9430700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d4060300 con 0x7f31d40f52d0 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:48762/0 (socket says 172.21.15.112:48762) 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d3fff700 1 -- 172.21.15.112:0/3514965124 learned_addr learned my addr 172.21.15.112:0/3514965124 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d3fff700 1 -- 172.21.15.112:0/3514965124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c8005040 con 0x7f31d40f52d0 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d3fff700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f31c80089f0 tx=0x7f31c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:13.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c8009320 con 0x7f31d40f52d0 2023-11-16T14:09:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d4060500 con 0x7f31d40f52d0 2023-11-16T14:09:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f31c8013ec0 con 0x7f31d40f52d0 2023-11-16T14:09:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c8014d30 con 0x7f31d40f52d0 2023-11-16T14:09:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.821+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d41099e0 con 0x7f31d40f52d0 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.822+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f31c80095f0 con 0x7f31d40f52d0 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.822+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d40591e0 con 0x7f31d40f52d0 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.823+0000 7f31d0ff9700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 0x7f31bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.823+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f31c8042460 con 0x7f31d40f52d0 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.823+0000 7f31d37fe700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 0x7f31bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:13.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.823+0000 7f31d37fe700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 0x7f31bc033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f31c40096a0 tx=0x7f31c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:13.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:13.828+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f31c801d7f0 con 0x7f31d40f52d0 2023-11-16T14:09:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:12.911914+0000 mgr.smithi061.zcjido (mgr.14156) 240 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:14.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.037+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31d4045bb0 con 0x7f31d40f52d0 2023-11-16T14:09:14.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.038+0000 7f31d0ff9700 1 -- 172.21.15.112:0/3514965124 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31c801d7f0 con 0x7f31d40f52d0 2023-11-16T14:09:14.040 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:14.040 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:14.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 msgr2=0x7f31bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:14.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 0x7f31bc033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f31c40096a0 tx=0x7f31c4006b40).stop 2023-11-16T14:09:14.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 msgr2=0x7f31d4109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:14.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f31c80089f0 tx=0x7f31c800b040).stop 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 shutdown_connections 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f31bc031090 0x7f31bc033550 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 --2- 172.21.15.112:0/3514965124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d40f52d0 0x7f31d4109330 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 >> 172.21.15.112:0/3514965124 conn(0x7f31d40f0a30 msgr2=0x7f31d40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.039+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 shutdown_connections 2023-11-16T14:09:14.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:14.040+0000 7f31d9430700 1 -- 172.21.15.112:0/3514965124 wait complete. 2023-11-16T14:09:14.044 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:15.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:14 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:14.038430+0000 mon.smithi061 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.112:0/3514965124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:15.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:15.548 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:16.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:14.912343+0000 mgr.smithi061.zcjido (mgr.14156) 241 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 -- 172.21.15.112:0/2803470852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 msgr2=0x7fce5c0f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 --2- 172.21.15.112:0/2803470852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c0f91f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fce4c009fb0 tx=0x7fce4c009b30).stop 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 -- 172.21.15.112:0/2803470852 shutdown_connections 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 --2- 172.21.15.112:0/2803470852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c0f91f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 -- 172.21.15.112:0/2803470852 >> 172.21.15.112:0/2803470852 conn(0x7fce5c0f0c10 msgr2=0x7fce5c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 -- 172.21.15.112:0/2803470852 shutdown_connections 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.676+0000 7fce622f3700 1 -- 172.21.15.112:0/2803470852 wait complete. 2023-11-16T14:09:16.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce622f3700 1 Processor -- start 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce622f3700 1 -- start start 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce622f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce622f3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c106c80 con 0x7fce5c060300 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54280/0 (socket says 172.21.15.112:54280) 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce5bfff700 1 -- 172.21.15.112:0/1783253827 learned_addr learned my addr 172.21.15.112:0/1783253827 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:16.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.677+0000 7fce5bfff700 1 -- 172.21.15.112:0/1783253827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c005040 con 0x7fce5c060300 2023-11-16T14:09:16.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce5bfff700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fce4c000c00 tx=0x7fce4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:16.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce4c0095f0 con 0x7fce5c060300 2023-11-16T14:09:16.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c106e80 con 0x7fce5c060300 2023-11-16T14:09:16.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fce4c007e50 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce4c014e70 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.678+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c107310 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fce4c025070 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce58ff9700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 0x7fce48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c045bb0 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fce4c041d10 con 0x7fce5c060300 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce5b7fe700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 0x7fce48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:16.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.679+0000 7fce5b7fe700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 0x7fce48033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fce440096a0 tx=0x7fce44006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:16.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.684+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fce4c0079c0 con 0x7fce5c060300 2023-11-16T14:09:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.891+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce5c108110 con 0x7fce5c060300 2023-11-16T14:09:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.892+0000 7fce58ff9700 1 -- 172.21.15.112:0/1783253827 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce4c00f300 con 0x7fce5c060300 2023-11-16T14:09:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:16.894 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 msgr2=0x7fce48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 0x7fce48033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fce440096a0 tx=0x7fce44006b40).stop 2023-11-16T14:09:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 msgr2=0x7fce5c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fce4c000c00 tx=0x7fce4c00b040).stop 2023-11-16T14:09:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 shutdown_connections 2023-11-16T14:09:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fce48031090 0x7fce48033550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 --2- 172.21.15.112:0/1783253827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5c060300 0x7fce5c106790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.893+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 >> 172.21.15.112:0/1783253827 conn(0x7fce5c0f0c10 msgr2=0x7fce5c0f16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.894+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 shutdown_connections 2023-11-16T14:09:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:16.894+0000 7fce622f3700 1 -- 172.21.15.112:0/1783253827 wait complete. 2023-11-16T14:09:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:17.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:16 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:16.892388+0000 mon.smithi061 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.112:0/1783253827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:16.912672+0000 mgr.smithi061.zcjido (mgr.14156) 242 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:18.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:18.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.363+0000 7fa108154700 1 -- 172.21.15.112:0/3254663464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 msgr2=0x7fa1000f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.363+0000 7fa108154700 1 --2- 172.21.15.112:0/3254663464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1000f58a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa0f0009fb0 tx=0x7fa0f0009b30).stop 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.363+0000 7fa108154700 1 -- 172.21.15.112:0/3254663464 shutdown_connections 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.363+0000 7fa108154700 1 --2- 172.21.15.112:0/3254663464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1000f58a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.363+0000 7fa108154700 1 -- 172.21.15.112:0/3254663464 >> 172.21.15.112:0/3254663464 conn(0x7fa1000f0be0 msgr2=0x7fa1000f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 -- 172.21.15.112:0/3254663464 shutdown_connections 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 -- 172.21.15.112:0/3254663464 wait complete. 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 Processor -- start 2023-11-16T14:09:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 -- start start 2023-11-16T14:09:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa108154700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100105be0 con 0x7fa1000f5480 2023-11-16T14:09:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa105ef0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa105ef0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54300/0 (socket says 172.21.15.112:54300) 2023-11-16T14:09:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.364+0000 7fa105ef0700 1 -- 172.21.15.112:0/135045965 learned_addr learned my addr 172.21.15.112:0/135045965 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa105ef0700 1 -- 172.21.15.112:0/135045965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0005040 con 0x7fa1000f5480 2023-11-16T14:09:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa105ef0700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa0f00089f0 tx=0x7fa0f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f0009320 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100105de0 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa0f0013ec0 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f0014d30 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.365+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa100106270 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.366+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa0f00095f0 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.366+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa100045bb0 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.366+0000 7fa0f67fc700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 0x7fa0ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.366+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa0f00424a0 con 0x7fa1000f5480 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.366+0000 7fa1056ef700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 0x7fa0ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:19.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.367+0000 7fa1056ef700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 0x7fa0ec033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0fc0096a0 tx=0x7fa0fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.371+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa0f001d7f0 con 0x7fa1000f5480 2023-11-16T14:09:19.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.579+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1000faf70 con 0x7fa1000f5480 2023-11-16T14:09:19.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.580+0000 7fa0f67fc700 1 -- 172.21.15.112:0/135045965 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0f001d7f0 con 0x7fa1000f5480 2023-11-16T14:09:19.582 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:19.582 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 msgr2=0x7fa0ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 0x7fa0ec033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0fc0096a0 tx=0x7fa0fc006b40).stop 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 msgr2=0x7fa1001056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa0f00089f0 tx=0x7fa0f000b040).stop 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 shutdown_connections 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa0ec031090 0x7fa0ec033550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 --2- 172.21.15.112:0/135045965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1000f5480 0x7fa1001056f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:19.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 >> 172.21.15.112:0/135045965 conn(0x7fa1000f0be0 msgr2=0x7fa1000f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:19.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.581+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 shutdown_connections 2023-11-16T14:09:19.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:19.582+0000 7fa108154700 1 -- 172.21.15.112:0/135045965 wait complete. 2023-11-16T14:09:19.585 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:18.912941+0000 mgr.smithi061.zcjido (mgr.14156) 243 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:19.580167+0000 mon.smithi061 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.112:0/135045965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:20.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:20.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.079+0000 7f8b3f566700 1 -- 172.21.15.112:0/2953769541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 msgr2=0x7f8b380f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.079+0000 7f8b3f566700 1 --2- 172.21.15.112:0/2953769541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b380f58a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8b28009fb0 tx=0x7f8b28009b30).stop 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.079+0000 7f8b3f566700 1 -- 172.21.15.112:0/2953769541 shutdown_connections 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.079+0000 7f8b3f566700 1 --2- 172.21.15.112:0/2953769541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b380f58a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.079+0000 7f8b3f566700 1 -- 172.21.15.112:0/2953769541 >> 172.21.15.112:0/2953769541 conn(0x7f8b380f0be0 msgr2=0x7f8b380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 -- 172.21.15.112:0/2953769541 shutdown_connections 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 -- 172.21.15.112:0/2953769541 wait complete. 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 Processor -- start 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 -- start start 2023-11-16T14:09:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3f566700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38105be0 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3d302700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3d302700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54316/0 (socket says 172.21.15.112:54316) 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.080+0000 7f8b3d302700 1 -- 172.21.15.112:0/3066484579 learned_addr learned my addr 172.21.15.112:0/3066484579 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b3d302700 1 -- 172.21.15.112:0/3066484579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b28005040 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b3d302700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8b280089f0 tx=0x7f8b2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b28009320 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8b28013ec0 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b38105de0 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b28014d30 con 0x7f8b380f5480 2023-11-16T14:09:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.081+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b38106270 con 0x7f8b380f5480 2023-11-16T14:09:22.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.082+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8b280095f0 con 0x7f8b380f5480 2023-11-16T14:09:22.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.082+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b38045bb0 con 0x7f8b380f5480 2023-11-16T14:09:22.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.082+0000 7f8b2dffb700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 0x7f8b24033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:22.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.082+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8b28044280 con 0x7f8b380f5480 2023-11-16T14:09:22.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.082+0000 7f8b3cb01700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 0x7f8b24033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:22.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.083+0000 7f8b3cb01700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 0x7f8b24033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8b340096a0 tx=0x7f8b34006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:22.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.087+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8b2801f980 con 0x7f8b380f5480 2023-11-16T14:09:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:20.913417+0000 mgr.smithi061.zcjido (mgr.14156) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:22.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.295+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b380faf70 con 0x7f8b380f5480 2023-11-16T14:09:22.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.296+0000 7f8b2dffb700 1 -- 172.21.15.112:0/3066484579 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b2801f380 con 0x7f8b380f5480 2023-11-16T14:09:22.298 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:22.298 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 msgr2=0x7f8b24033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 0x7f8b24033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8b340096a0 tx=0x7f8b34006b40).stop 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 msgr2=0x7f8b381056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8b280089f0 tx=0x7f8b2800b040).stop 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 shutdown_connections 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f8b24031090 0x7f8b24033550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 --2- 172.21.15.112:0/3066484579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b380f5480 0x7f8b381056f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 >> 172.21.15.112:0/3066484579 conn(0x7f8b380f0be0 msgr2=0x7f8b380f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 shutdown_connections 2023-11-16T14:09:22.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:22.298+0000 7f8b3f566700 1 -- 172.21.15.112:0/3066484579 wait complete. 2023-11-16T14:09:22.301 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:23.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:22 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:22.296538+0000 mon.smithi061 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.112:0/3066484579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:23.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:23.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:24.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:22.913833+0000 mgr.smithi061.zcjido (mgr.14156) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:24.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.750+0000 7fad0b59e700 1 -- 172.21.15.112:0/2568118069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 msgr2=0x7fad0c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:24.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.750+0000 7fad0b59e700 1 --2- 172.21.15.112:0/2568118069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c0f56f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7facf4009fb0 tx=0x7facf4009b30).stop 2023-11-16T14:09:24.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- 172.21.15.112:0/2568118069 shutdown_connections 2023-11-16T14:09:24.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 --2- 172.21.15.112:0/2568118069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c0f56f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- 172.21.15.112:0/2568118069 >> 172.21.15.112:0/2568118069 conn(0x7fad0c0f0a30 msgr2=0x7fad0c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- 172.21.15.112:0/2568118069 shutdown_connections 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- 172.21.15.112:0/2568118069 wait complete. 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 Processor -- start 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- start start 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:24.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.751+0000 7fad0b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c106b10 con 0x7fad0c0f52d0 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:54328/0 (socket says 172.21.15.112:54328) 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0a59c700 1 -- 172.21.15.112:0/3639588807 learned_addr learned my addr 172.21.15.112:0/3639588807 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0a59c700 1 -- 172.21.15.112:0/3639588807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4005040 con 0x7fad0c0f52d0 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0a59c700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7facf40089f0 tx=0x7facf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf4009320 con 0x7fad0c0f52d0 2023-11-16T14:09:24.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7facf4013ec0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0c106d10 con 0x7fad0c0f52d0 2023-11-16T14:09:24.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf4014d30 con 0x7fad0c0f52d0 2023-11-16T14:09:24.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.752+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0c1071a0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.753+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7facf40095f0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.753+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0c045bb0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.753+0000 7fad02ffd700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 0x7facf8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.753+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7facf40423e0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.753+0000 7fad09d9b700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 0x7facf8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:24.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.754+0000 7fad09d9b700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 0x7facf8033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7facfc0096a0 tx=0x7facfc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:24.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.758+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7facf401d7f0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.964+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad0c108050 con 0x7fad0c0f52d0 2023-11-16T14:09:24.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.965+0000 7fad02ffd700 1 -- 172.21.15.112:0/3639588807 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facf401d7f0 con 0x7fad0c0f52d0 2023-11-16T14:09:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:24.968 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:24.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 msgr2=0x7facf8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:24.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 0x7facf8033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7facfc0096a0 tx=0x7facfc006b40).stop 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 msgr2=0x7fad0c106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7facf40089f0 tx=0x7facf400b040).stop 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 shutdown_connections 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7facf8031090 0x7facf8033550 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 --2- 172.21.15.112:0/3639588807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad0c0f52d0 0x7fad0c106620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 >> 172.21.15.112:0/3639588807 conn(0x7fad0c0f0a30 msgr2=0x7fad0c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 shutdown_connections 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:24.967+0000 7fad0b59e700 1 -- 172.21.15.112:0/3639588807 wait complete. 2023-11-16T14:09:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:25.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:24.965679+0000 mon.smithi061 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.112:0/3639588807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:26.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:25 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:24.914212+0000 mgr.smithi061.zcjido (mgr.14156) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:26.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:26.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 -- 172.21.15.112:0/2968890698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 msgr2=0x7f6ffc0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 --2- 172.21.15.112:0/2968890698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc0f58d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6fe4009fb0 tx=0x7f6fe4009b30).stop 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 -- 172.21.15.112:0/2968890698 shutdown_connections 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 --2- 172.21.15.112:0/2968890698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc0f58d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 -- 172.21.15.112:0/2968890698 >> 172.21.15.112:0/2968890698 conn(0x7f6ffc0f0c10 msgr2=0x7f6ffc0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 -- 172.21.15.112:0/2968890698 shutdown_connections 2023-11-16T14:09:27.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.449+0000 7f7002724700 1 -- 172.21.15.112:0/2968890698 wait complete. 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f7002724700 1 Processor -- start 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f7002724700 1 -- start start 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f7002724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f7002724700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc107300 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f6ffbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f6ffbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53192/0 (socket says 172.21.15.112:53192) 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f6ffbfff700 1 -- 172.21.15.112:0/1918573633 learned_addr learned my addr 172.21.15.112:0/1918573633 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f6ffbfff700 1 -- 172.21.15.112:0/1918573633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4005040 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.450+0000 7f6ffbfff700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f6fe40089f0 tx=0x7f6fe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:27.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.451+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe4009320 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.451+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc060300 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.451+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6fe4013ec0 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.451+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe4014d30 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.451+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc10be00 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.452+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6fe4014420 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.452+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc045bb0 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.452+0000 7f6ff8ff9700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 0x7f6fe80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.452+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6fe40423b0 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.452+0000 7f6ffb7fe700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 0x7f6fe80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:27.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.453+0000 7f6ffb7fe700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 0x7f6fe80335a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6fec0096a0 tx=0x7f6fec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:27.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.457+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6fe401d980 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.666+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ffc0fafa0 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.667+0000 7f6ff8ff9700 1 -- 172.21.15.112:0/1918573633 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fe401d380 con 0x7f6ffc0f54b0 2023-11-16T14:09:27.670 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:27.670 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:27.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.668+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 msgr2=0x7f6fe80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.668+0000 7f7002724700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 0x7f6fe80335a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6fec0096a0 tx=0x7f6fec006b40).stop 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.668+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 msgr2=0x7f6ffc10b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.668+0000 7f7002724700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f6fe40089f0 tx=0x7f6fe400b040).stop 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 shutdown_connections 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6fe80310e0 0x7f6fe80335a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 --2- 172.21.15.112:0/1918573633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ffc0f54b0 0x7f6ffc10b910 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 >> 172.21.15.112:0/1918573633 conn(0x7f6ffc0f0c10 msgr2=0x7f6ffc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 shutdown_connections 2023-11-16T14:09:27.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:27.669+0000 7f7002724700 1 -- 172.21.15.112:0/1918573633 wait complete. 2023-11-16T14:09:27.673 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:26.914576+0000 mgr.smithi061.zcjido (mgr.14156) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:28.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:27 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:27.668214+0000 mon.smithi061 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.112:0/1918573633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:29.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:29.025 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:30.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 -- 172.21.15.112:0/437904583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 msgr2=0x7f4f100f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:30.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 --2- 172.21.15.112:0/437904583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f100f69b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f4f04009fb0 tx=0x7f4f04009b30).stop 2023-11-16T14:09:30.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 -- 172.21.15.112:0/437904583 shutdown_connections 2023-11-16T14:09:30.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 --2- 172.21.15.112:0/437904583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f100f69b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:30.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 -- 172.21.15.112:0/437904583 >> 172.21.15.112:0/437904583 conn(0x7f4f100f1cf0 msgr2=0x7f4f100f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 -- 172.21.15.112:0/437904583 shutdown_connections 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.143+0000 7f4f1569b700 1 -- 172.21.15.112:0/437904583 wait complete. 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.144+0000 7f4f1569b700 1 Processor -- start 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.144+0000 7f4f1569b700 1 -- start start 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.144+0000 7f4f1569b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:30.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.144+0000 7f4f1569b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f10106cf0 con 0x7f4f100f6590 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53210/0 (socket says 172.21.15.112:53210) 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f0effd700 1 -- 172.21.15.112:0/36906710 learned_addr learned my addr 172.21.15.112:0/36906710 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f0effd700 1 -- 172.21.15.112:0/36906710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04005040 con 0x7f4f100f6590 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f0effd700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f4f04000c00 tx=0x7f4f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f040095f0 con 0x7f4f100f6590 2023-11-16T14:09:30.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f10106ef0 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4f04007e50 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f04014e70 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.145+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f10107380 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.146+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4f04025070 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.146+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f10045bb0 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.146+0000 7f4ef77fe700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.146+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4f04041de0 con 0x7f4f100f6590 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.147+0000 7f4f0e7fc700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:30.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.147+0000 7f4f0e7fc700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4f000096a0 tx=0x7f4f00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:30.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.151+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4f0400f8d0 con 0x7f4f100f6590 2023-11-16T14:09:30.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:29 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:28.914914+0000 mgr.smithi061.zcjido (mgr.14156) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:30.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.360+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f100fc080 con 0x7f4f100f6590 2023-11-16T14:09:30.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.361+0000 7f4ef77fe700 1 -- 172.21.15.112:0/36906710 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f0400f300 con 0x7f4f100f6590 2023-11-16T14:09:30.364 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:30.364 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 msgr2=0x7f4ef8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4f000096a0 tx=0x7f4f00006b40).stop 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 msgr2=0x7f4f10106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f4f04000c00 tx=0x7f4f0400b040).stop 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 shutdown_connections 2023-11-16T14:09:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:30.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 --2- 172.21.15.112:0/36906710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f100f6590 0x7f4f10106800 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:30.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 >> 172.21.15.112:0/36906710 conn(0x7f4f100f1cf0 msgr2=0x7f4f100f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:30.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 shutdown_connections 2023-11-16T14:09:30.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:30.362+0000 7f4f1569b700 1 -- 172.21.15.112:0/36906710 wait complete. 2023-11-16T14:09:30.366 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:31.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:30 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:30.362296+0000 mon.smithi061 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.112:0/36906710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:31.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:31.731 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:32.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:31 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:30.915329+0000 mgr.smithi061.zcjido (mgr.14156) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:32.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 -- 172.21.15.112:0/2499543398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 msgr2=0x7fa3340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2499543398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa3340f69b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fa330004760 tx=0x7fa330009ad0).stop 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 -- 172.21.15.112:0/2499543398 shutdown_connections 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2499543398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa3340f69b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 -- 172.21.15.112:0/2499543398 >> 172.21.15.112:0/2499543398 conn(0x7fa3340f1cd0 msgr2=0x7fa3340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 -- 172.21.15.112:0/2499543398 shutdown_connections 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.852+0000 7fa33c58a700 1 -- 172.21.15.112:0/2499543398 wait complete. 2023-11-16T14:09:32.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33c58a700 1 Processor -- start 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33c58a700 1 -- start start 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33c58a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33c58a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334069f00 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33a326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33a326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:53224/0 (socket says 172.21.15.112:53224) 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33a326700 1 -- 172.21.15.112:0/2260497636 learned_addr learned my addr 172.21.15.112:0/2260497636 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33a326700 1 -- 172.21.15.112:0/2260497636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa330009d70 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa33a326700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fa330004170 tx=0x7fa330009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.853+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33000e040 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33406a0a0 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa330008ef0 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa330014c90 con 0x7fa3340f6590 2023-11-16T14:09:32.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33406a530 con 0x7fa3340f6590 2023-11-16T14:09:32.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa330014420 con 0x7fa3340f6590 2023-11-16T14:09:32.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.854+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3340591e0 con 0x7fa3340f6590 2023-11-16T14:09:32.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.855+0000 7fa32affd700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 0x7fa320033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:32.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.855+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa3300422b0 con 0x7fa3340f6590 2023-11-16T14:09:32.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.855+0000 7fa339b25700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 0x7fa320033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:32.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.855+0000 7fa339b25700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 0x7fa320033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa3240096a0 tx=0x7fa324006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:32.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:32.860+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa330026300 con 0x7fa3340f6590 2023-11-16T14:09:33.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.070+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3340fc080 con 0x7fa3340f6590 2023-11-16T14:09:33.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.070+0000 7fa32affd700 1 -- 172.21.15.112:0/2260497636 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa330012b20 con 0x7fa3340f6590 2023-11-16T14:09:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:33.074 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:33.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 msgr2=0x7fa320033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:33.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 0x7fa320033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa3240096a0 tx=0x7fa324006b40).stop 2023-11-16T14:09:33.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 msgr2=0x7fa33406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:33.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fa330004170 tx=0x7fa330009ad0).stop 2023-11-16T14:09:33.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 shutdown_connections 2023-11-16T14:09:33.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fa320031090 0x7fa320033550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:33.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 --2- 172.21.15.112:0/2260497636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340f6590 0x7fa33406ba80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:33.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 >> 172.21.15.112:0/2260497636 conn(0x7fa3340f1cd0 msgr2=0x7fa3340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:33.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 shutdown_connections 2023-11-16T14:09:33.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:33.072+0000 7fa33c58a700 1 -- 172.21.15.112:0/2260497636 wait complete. 2023-11-16T14:09:33.077 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:32.915599+0000 mgr.smithi061.zcjido (mgr.14156) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:34.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:33 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:33.072009+0000 mon.smithi061 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.112:0/2260497636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:34.439 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:34.439 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:35.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.514+0000 7fe51c35e700 1 -- 172.21.15.112:0/3876260231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 msgr2=0x7fe5140f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.514+0000 7fe51c35e700 1 --2- 172.21.15.112:0/3876260231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5140f4c70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe510009fb0 tx=0x7fe510009b30).stop 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 -- 172.21.15.112:0/3876260231 shutdown_connections 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 --2- 172.21.15.112:0/3876260231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5140f4c70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 -- 172.21.15.112:0/3876260231 >> 172.21.15.112:0/3876260231 conn(0x7fe5140f0390 msgr2=0x7fe5140f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 -- 172.21.15.112:0/3876260231 shutdown_connections 2023-11-16T14:09:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 -- 172.21.15.112:0/3876260231 wait complete. 2023-11-16T14:09:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.515+0000 7fe51c35e700 1 Processor -- start 2023-11-16T14:09:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51c35e700 1 -- start start 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51c35e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51c35e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe514106c90 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51a0fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51a0fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44010/0 (socket says 172.21.15.112:44010) 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51a0fa700 1 -- 172.21.15.112:0/907245476 learned_addr learned my addr 172.21.15.112:0/907245476 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51a0fa700 1 -- 172.21.15.112:0/907245476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510005040 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.516+0000 7fe51a0fa700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fe510005010 tx=0x7fe51000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.517+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5100095f0 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.517+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe514106e90 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.517+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe510007e50 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.517+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe510014de0 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.517+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe514107320 con 0x7fe5140f4850 2023-11-16T14:09:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe510025070 con 0x7fe5140f4850 2023-11-16T14:09:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe514045bb0 con 0x7fe5140f4850 2023-11-16T14:09:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe50affd700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 0x7fe500033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe510041eb0 con 0x7fe5140f4850 2023-11-16T14:09:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe5198f9700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 0x7fe500033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.518+0000 7fe5198f9700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 0x7fe500033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.522+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe51000f4e0 con 0x7fe5140f4850 2023-11-16T14:09:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.730+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5141081d0 con 0x7fe5140f4850 2023-11-16T14:09:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.731+0000 7fe50affd700 1 -- 172.21.15.112:0/907245476 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe51000f300 con 0x7fe5140f4850 2023-11-16T14:09:35.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:35.734 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:35.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.732+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 msgr2=0x7fe500033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:35.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.732+0000 7fe51c35e700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 0x7fe500033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).stop 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.732+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 msgr2=0x7fe5141067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.732+0000 7fe51c35e700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fe510005010 tx=0x7fe51000b040).stop 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 shutdown_connections 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe500031090 0x7fe500033550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 --2- 172.21.15.112:0/907245476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5140f4850 0x7fe5141067a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 >> 172.21.15.112:0/907245476 conn(0x7fe5140f0390 msgr2=0x7fe5140f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 shutdown_connections 2023-11-16T14:09:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:35.733+0000 7fe51c35e700 1 -- 172.21.15.112:0/907245476 wait complete. 2023-11-16T14:09:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:34.915924+0000 mgr.smithi061.zcjido (mgr.14156) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:34.985762+0000 mon.smithi061 (mon.0) 295 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/mirror_snapshot_schedule"}]: dispatch 2023-11-16T14:09:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:34.985971+0000 mon.smithi061 (mon.0) 296 : audit [INF] from='mgr.14156 172.21.15.61:0/789355747' entity='mgr.smithi061.zcjido' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.zcjido/trash_purge_schedule"}]: dispatch 2023-11-16T14:09:36.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:35 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:35.732499+0000 mon.smithi061 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.112:0/907245476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:37.112 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:37.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 -- 172.21.15.112:0/3857127530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 msgr2=0x7f01bc0fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 --2- 172.21.15.112:0/3857127530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc0fb5e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f01a4009fb0 tx=0x7f01a4009b30).stop 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 -- 172.21.15.112:0/3857127530 shutdown_connections 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 --2- 172.21.15.112:0/3857127530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc0fb5e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 -- 172.21.15.112:0/3857127530 >> 172.21.15.112:0/3857127530 conn(0x7f01bc0f0be0 msgr2=0x7f01bc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 -- 172.21.15.112:0/3857127530 shutdown_connections 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 -- 172.21.15.112:0/3857127530 wait complete. 2023-11-16T14:09:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.235+0000 7f01c1f38700 1 Processor -- start 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01c1f38700 1 -- start start 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01c1f38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01c1f38700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc108e60 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44030/0 (socket says 172.21.15.112:44030) 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01bb7fe700 1 -- 172.21.15.112:0/2164964416 learned_addr learned my addr 172.21.15.112:0/2164964416 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01bb7fe700 1 -- 172.21.15.112:0/2164964416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4005040 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.236+0000 7f01bb7fe700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f01a4009f80 tx=0x7f01a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a4009320 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc109060 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f01a4013680 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a401eda0 con 0x7f01bc0602e0 2023-11-16T14:09:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc1094f0 con 0x7f01bc0602e0 2023-11-16T14:09:38.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.237+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f01a401e420 con 0x7f01bc0602e0 2023-11-16T14:09:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.238+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc045bb0 con 0x7f01bc0602e0 2023-11-16T14:09:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.238+0000 7f01b3fff700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 0x7f01a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.238+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f01a4042360 con 0x7f01bc0602e0 2023-11-16T14:09:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.238+0000 7f01baffd700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 0x7f01a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.238+0000 7f01baffd700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 0x7f01a8033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f01ac0096a0 tx=0x7f01ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:38.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.243+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f01a401d380 con 0x7f01bc0602e0 2023-11-16T14:09:38.257 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:37 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:36.916191+0000 mgr.smithi061.zcjido (mgr.14156) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.451+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01bc10a340 con 0x7f01bc0602e0 2023-11-16T14:09:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.452+0000 7f01b3fff700 1 -- 172.21.15.112:0/2164964416 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01a401cb10 con 0x7f01bc0602e0 2023-11-16T14:09:38.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:38.455 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 msgr2=0x7f01a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 0x7f01a8033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f01ac0096a0 tx=0x7f01ac006b40).stop 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 msgr2=0x7f01bc108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f01a4009f80 tx=0x7f01a400b040).stop 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 shutdown_connections 2023-11-16T14:09:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f01a8031090 0x7f01a8033550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 --2- 172.21.15.112:0/2164964416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01bc0602e0 0x7f01bc108970 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 >> 172.21.15.112:0/2164964416 conn(0x7f01bc0f0be0 msgr2=0x7f01bc0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 shutdown_connections 2023-11-16T14:09:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:38.453+0000 7f01c1f38700 1 -- 172.21.15.112:0/2164964416 wait complete. 2023-11-16T14:09:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:39.257 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:38 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:38.453292+0000 mon.smithi061 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.112:0/2164964416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:39.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:39.875 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:40.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:39 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:38.916528+0000 mgr.smithi061.zcjido (mgr.14156) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 -- 172.21.15.112:0/4232805351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 msgr2=0x7f9a840f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4232805351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a840f91c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f9a78009fb0 tx=0x7f9a78009b30).stop 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 -- 172.21.15.112:0/4232805351 shutdown_connections 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4232805351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a840f91c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 -- 172.21.15.112:0/4232805351 >> 172.21.15.112:0/4232805351 conn(0x7f9a840f0be0 msgr2=0x7f9a840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 -- 172.21.15.112:0/4232805351 shutdown_connections 2023-11-16T14:09:40.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.991+0000 7f9a8972c700 1 -- 172.21.15.112:0/4232805351 wait complete. 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a8972c700 1 Processor -- start 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a8972c700 1 -- start start 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a8972c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a8972c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a84106c70 con 0x7f9a84060300 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:40.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44046/0 (socket says 172.21.15.112:44046) 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a82ffd700 1 -- 172.21.15.112:0/4169405120 learned_addr learned my addr 172.21.15.112:0/4169405120 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a82ffd700 1 -- 172.21.15.112:0/4169405120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78005040 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.992+0000 7f9a82ffd700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f9a78000c00 tx=0x7f9a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a780095f0 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a84106e70 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a78007e50 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a78014e70 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a84107300 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.993+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9a78025070 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.994+0000 7f9a6b7fe700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 0x7f9a6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.994+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a84045bb0 con 0x7f9a84060300 2023-11-16T14:09:40.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.994+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a78041d90 con 0x7f9a84060300 2023-11-16T14:09:40.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.994+0000 7f9a827fc700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 0x7f9a6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:40.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.994+0000 7f9a827fc700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 0x7f9a6c033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:41.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:40.999+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a780079c0 con 0x7f9a84060300 2023-11-16T14:09:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.207+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a84108100 con 0x7f9a84060300 2023-11-16T14:09:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.207+0000 7f9a6b7fe700 1 -- 172.21.15.112:0/4169405120 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a7802d090 con 0x7f9a84060300 2023-11-16T14:09:41.210 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:41.211 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 msgr2=0x7f9a6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 0x7f9a6c033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).stop 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 msgr2=0x7f9a84106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f9a78000c00 tx=0x7f9a7800b040).stop 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 shutdown_connections 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f9a6c031090 0x7f9a6c033550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 --2- 172.21.15.112:0/4169405120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a84060300 0x7f9a84106780 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 >> 172.21.15.112:0/4169405120 conn(0x7f9a840f0be0 msgr2=0x7f9a840f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 shutdown_connections 2023-11-16T14:09:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:41.209+0000 7f9a8972c700 1 -- 172.21.15.112:0/4169405120 wait complete. 2023-11-16T14:09:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:40.916901+0000 mgr.smithi061.zcjido (mgr.14156) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:42.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:41 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:41.209124+0000 mon.smithi061 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.112:0/4169405120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:42.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:42.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:43.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 -- 172.21.15.112:0/3600344464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 msgr2=0x7fcc800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 --2- 172.21.15.112:0/3600344464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc800f69b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fcc68004760 tx=0x7fcc68009ad0).stop 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 -- 172.21.15.112:0/3600344464 shutdown_connections 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 --2- 172.21.15.112:0/3600344464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc800f69b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 -- 172.21.15.112:0/3600344464 >> 172.21.15.112:0/3600344464 conn(0x7fcc800f1cd0 msgr2=0x7fcc800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 -- 172.21.15.112:0/3600344464 shutdown_connections 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.713+0000 7fcc86a44700 1 -- 172.21.15.112:0/3600344464 wait complete. 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc86a44700 1 Processor -- start 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc86a44700 1 -- start start 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc86a44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:43.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc86a44700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80106cb0 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44062/0 (socket says 172.21.15.112:44062) 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc7ffff700 1 -- 172.21.15.112:0/2529243934 learned_addr learned my addr 172.21.15.112:0/2529243934 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.714+0000 7fcc7ffff700 1 -- 172.21.15.112:0/2529243934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68009d70 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc7ffff700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fcc68004170 tx=0x7fcc68009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc6800e040 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc80106eb0 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcc68008ef0 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc68014c90 con 0x7fcc800f6590 2023-11-16T14:09:43.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.715+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc80107340 con 0x7fcc800f6590 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcc68014420 con 0x7fcc800f6590 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc80045bb0 con 0x7fcc800f6590 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc7cff9700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 0x7fcc6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcc680422b0 con 0x7fcc800f6590 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc7f7fe700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 0x7fcc6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.716+0000 7fcc7f7fe700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 0x7fcc6c033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcc700096a0 tx=0x7fcc70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:43.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.721+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcc68012d00 con 0x7fcc800f6590 2023-11-16T14:09:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.930+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc801081f0 con 0x7fcc800f6590 2023-11-16T14:09:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.931+0000 7fcc7cff9700 1 -- 172.21.15.112:0/2529243934 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc68012b20 con 0x7fcc800f6590 2023-11-16T14:09:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:43.934 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:43.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 msgr2=0x7fcc6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:43.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 0x7fcc6c033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcc700096a0 tx=0x7fcc70006b40).stop 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 msgr2=0x7fcc801067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fcc68004170 tx=0x7fcc68009ad0).stop 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 shutdown_connections 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fcc6c031090 0x7fcc6c033550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 --2- 172.21.15.112:0/2529243934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc800f6590 0x7fcc801067c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.932+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 >> 172.21.15.112:0/2529243934 conn(0x7fcc800f1cd0 msgr2=0x7fcc800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.933+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 shutdown_connections 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:43.933+0000 7fcc86a44700 1 -- 172.21.15.112:0/2529243934 wait complete. 2023-11-16T14:09:43.936 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:44.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:43 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:42.917344+0000 mgr.smithi061.zcjido (mgr.14156) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:45.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:44 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:43.931999+0000 mon.smithi061 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.112:0/2529243934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:45.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:45.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:46.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:45 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:44.917752+0000 mgr.smithi061.zcjido (mgr.14156) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 -- 172.21.15.112:0/2587654442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 msgr2=0x7fca380f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 --2- 172.21.15.112:0/2587654442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca380f9160 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fca20009fb0 tx=0x7fca20009b30).stop 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 -- 172.21.15.112:0/2587654442 shutdown_connections 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 --2- 172.21.15.112:0/2587654442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca380f9160 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 -- 172.21.15.112:0/2587654442 >> 172.21.15.112:0/2587654442 conn(0x7fca380f0be0 msgr2=0x7fca380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 -- 172.21.15.112:0/2587654442 shutdown_connections 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.374+0000 7fca3db38700 1 -- 172.21.15.112:0/2587654442 wait complete. 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca3db38700 1 Processor -- start 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca3db38700 1 -- start start 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca3db38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca3db38700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38106c40 con 0x7fca38060360 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34236/0 (socket says 172.21.15.112:34236) 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca377fe700 1 -- 172.21.15.112:0/3958599547 learned_addr learned my addr 172.21.15.112:0/3958599547 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca377fe700 1 -- 172.21.15.112:0/3958599547 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca20005040 con 0x7fca38060360 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.375+0000 7fca377fe700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fca20009f80 tx=0x7fca2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.376+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca20009320 con 0x7fca38060360 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.376+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca20013680 con 0x7fca38060360 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.376+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca38106e40 con 0x7fca38060360 2023-11-16T14:09:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.376+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca2001eda0 con 0x7fca38060360 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.376+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca381072d0 con 0x7fca38060360 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.377+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fca2001e420 con 0x7fca38060360 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.377+0000 7fca2ffff700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 0x7fca24033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.377+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca38045bb0 con 0x7fca38060360 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.377+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fca20042350 con 0x7fca38060360 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.377+0000 7fca36ffd700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 0x7fca24033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.378+0000 7fca36ffd700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 0x7fca24033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.382+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca2001e700 con 0x7fca38060360 2023-11-16T14:09:46.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.589+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca381080d0 con 0x7fca38060360 2023-11-16T14:09:46.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.590+0000 7fca2ffff700 1 -- 172.21.15.112:0/3958599547 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca2001cb10 con 0x7fca38060360 2023-11-16T14:09:46.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:46.595 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:46.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 msgr2=0x7fca24033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:46.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 0x7fca24033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).stop 2023-11-16T14:09:46.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 msgr2=0x7fca38106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fca20009f80 tx=0x7fca2000b040).stop 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 shutdown_connections 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fca24031090 0x7fca24033550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 --2- 172.21.15.112:0/3958599547 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca38060360 0x7fca38106750 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 >> 172.21.15.112:0/3958599547 conn(0x7fca380f0be0 msgr2=0x7fca380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 shutdown_connections 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:46.592+0000 7fca3db38700 1 -- 172.21.15.112:0/3958599547 wait complete. 2023-11-16T14:09:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:47.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:46 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:46.591784+0000 mon.smithi061 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.112:0/3958599547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:47.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:47.970 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:48.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:47 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:46.918077+0000 mgr.smithi061.zcjido (mgr.14156) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:49.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 -- 172.21.15.112:0/4211174298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 msgr2=0x7fe08c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:49.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 --2- 172.21.15.112:0/4211174298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c0f58a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fe07c009fb0 tx=0x7fe07c009b30).stop 2023-11-16T14:09:49.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 -- 172.21.15.112:0/4211174298 shutdown_connections 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 --2- 172.21.15.112:0/4211174298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c0f58a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 -- 172.21.15.112:0/4211174298 >> 172.21.15.112:0/4211174298 conn(0x7fe08c0f0be0 msgr2=0x7fe08c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 -- 172.21.15.112:0/4211174298 shutdown_connections 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.096+0000 7fe090996700 1 -- 172.21.15.112:0/4211174298 wait complete. 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe090996700 1 Processor -- start 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe090996700 1 -- start start 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe090996700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe090996700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c106cd0 con 0x7fe08c0f5480 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe08a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe08a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34258/0 (socket says 172.21.15.112:34258) 2023-11-16T14:09:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe08a59c700 1 -- 172.21.15.112:0/1987824762 learned_addr learned my addr 172.21.15.112:0/1987824762 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe08a59c700 1 -- 172.21.15.112:0/1987824762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c005040 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe08a59c700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe07c0089f0 tx=0x7fe07c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07c009320 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.097+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c106ed0 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe07c013ec0 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07c014d30 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c107360 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe07c014420 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe07affd700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 0x7fe074033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.098+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe07c042460 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.099+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c045bb0 con 0x7fe08c0f5480 2023-11-16T14:09:49.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.099+0000 7fe089d9b700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 0x7fe074033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:49.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.099+0000 7fe089d9b700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 0x7fe074033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:49.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.105+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe07c014750 con 0x7fe08c0f5480 2023-11-16T14:09:49.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.313+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe08c108080 con 0x7fe08c0f5480 2023-11-16T14:09:49.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe07affd700 1 -- 172.21.15.112:0/1987824762 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe07c01d7f0 con 0x7fe08c0f5480 2023-11-16T14:09:49.316 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:49.316 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:49.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 msgr2=0x7fe074033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:49.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 0x7fe074033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).stop 2023-11-16T14:09:49.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 msgr2=0x7fe08c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:49.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe07c0089f0 tx=0x7fe07c00b040).stop 2023-11-16T14:09:49.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 shutdown_connections 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fe074031090 0x7fe074033550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 --2- 172.21.15.112:0/1987824762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe08c0f5480 0x7fe08c1067e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 >> 172.21.15.112:0/1987824762 conn(0x7fe08c0f0be0 msgr2=0x7fe08c05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 shutdown_connections 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:49.315+0000 7fe090996700 1 -- 172.21.15.112:0/1987824762 wait complete. 2023-11-16T14:09:49.319 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:48.918426+0000 mgr.smithi061.zcjido (mgr.14156) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:50.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:49 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:49.315188+0000 mon.smithi061 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.112:0/1987824762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:50.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:50.642 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 -- 172.21.15.112:0/3465176937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 msgr2=0x7f4f0c0f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/3465176937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c0f9200 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4ef4009fb0 tx=0x7f4ef4009b30).stop 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 -- 172.21.15.112:0/3465176937 shutdown_connections 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/3465176937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c0f9200 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 -- 172.21.15.112:0/3465176937 >> 172.21.15.112:0/3465176937 conn(0x7f4f0c0f0be0 msgr2=0x7f4f0c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 -- 172.21.15.112:0/3465176937 shutdown_connections 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.714+0000 7f4f11c1a700 1 -- 172.21.15.112:0/3465176937 wait complete. 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f11c1a700 1 Processor -- start 2023-11-16T14:09:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f11c1a700 1 -- start start 2023-11-16T14:09:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f11c1a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f11c1a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c106c20 con 0x7f4f0c0602c0 2023-11-16T14:09:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34284/0 (socket says 172.21.15.112:34284) 2023-11-16T14:09:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.715+0000 7f4f0b7fe700 1 -- 172.21.15.112:0/2730634195 learned_addr learned my addr 172.21.15.112:0/2730634195 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f0b7fe700 1 -- 172.21.15.112:0/2730634195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef4005040 con 0x7f4f0c0602c0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f0b7fe700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4ef4013fd0 tx=0x7f4ef4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef4018070 con 0x7f4f0c0602c0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c106e20 con 0x7f4f0c0602c0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4ef4007b10 con 0x7f4f0c0602c0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ef401c3c0 con 0x7f4f0c0602c0 2023-11-16T14:09:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.716+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c1072b0 con 0x7f4f0c0602c0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4ef401c640 con 0x7f4f0c0602c0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f03fff700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4ef4042a80 con 0x7f4f0c0602c0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c0591e0 con 0x7f4f0c0602c0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f0affd700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.717+0000 7f4f0affd700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4efc0096a0 tx=0x7f4efc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.722+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4ef4024300 con 0x7f4f0c0602c0 2023-11-16T14:09:51.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.930+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f0c045bb0 con 0x7f4f0c0602c0 2023-11-16T14:09:51.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.931+0000 7f4f03fff700 1 -- 172.21.15.112:0/2730634195 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ef401bb60 con 0x7f4f0c0602c0 2023-11-16T14:09:51.932 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:51.933 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 msgr2=0x7f4ef8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4efc0096a0 tx=0x7f4efc006b40).stop 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 msgr2=0x7f4f0c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4ef4013fd0 tx=0x7f4ef4009b30).stop 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 shutdown_connections 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f4ef8031090 0x7f4ef8033550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 --2- 172.21.15.112:0/2730634195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f0c0602c0 0x7f4f0c106730 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 >> 172.21.15.112:0/2730634195 conn(0x7f4f0c0f0be0 msgr2=0x7f4f0c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 shutdown_connections 2023-11-16T14:09:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:51.932+0000 7f4f11c1a700 1 -- 172.21.15.112:0/2730634195 wait complete. 2023-11-16T14:09:51.935 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:50.918808+0000 mgr.smithi061.zcjido (mgr.14156) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:52.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:51 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:51.931189+0000 mon.smithi061 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.112:0/2730634195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:53.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:53.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:54.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:53 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:52.919204+0000 mgr.smithi061.zcjido (mgr.14156) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:54.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 -- 172.21.15.112:0/573275313 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 msgr2=0x7f59f00f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:54.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 --2- 172.21.15.112:0/573275313 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f00f91c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f59e0009fb0 tx=0x7f59e0009b30).stop 2023-11-16T14:09:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 -- 172.21.15.112:0/573275313 shutdown_connections 2023-11-16T14:09:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 --2- 172.21.15.112:0/573275313 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f00f91c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 -- 172.21.15.112:0/573275313 >> 172.21.15.112:0/573275313 conn(0x7f59f00f0be0 msgr2=0x7f59f00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 -- 172.21.15.112:0/573275313 shutdown_connections 2023-11-16T14:09:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.470+0000 7f59f656c700 1 -- 172.21.15.112:0/573275313 wait complete. 2023-11-16T14:09:54.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.471+0000 7f59f656c700 1 Processor -- start 2023-11-16T14:09:54.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.471+0000 7f59f656c700 1 -- start start 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.471+0000 7f59f656c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.471+0000 7f59f656c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f0106c70 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59effff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59effff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:34312/0 (socket says 172.21.15.112:34312) 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59effff700 1 -- 172.21.15.112:0/2259564241 learned_addr learned my addr 172.21.15.112:0/2259564241 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59effff700 1 -- 172.21.15.112:0/2259564241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e0005040 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59effff700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f59e0000c00 tx=0x7f59e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e00095f0 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f59e0007e50 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f0106e70 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f0107300 con 0x7f59f0060300 2023-11-16T14:09:54.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.472+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e0014e70 con 0x7f59f0060300 2023-11-16T14:09:54.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.473+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f59e0025070 con 0x7f59f0060300 2023-11-16T14:09:54.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.473+0000 7f59ecff9700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 0x7f59dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:09:54.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.473+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f59e0041df0 con 0x7f59f0060300 2023-11-16T14:09:54.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.473+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f0045bb0 con 0x7f59f0060300 2023-11-16T14:09:54.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.473+0000 7f59ef7fe700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 0x7f59dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:09:54.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.474+0000 7f59ef7fe700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 0x7f59dc033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f59d80096a0 tx=0x7f59d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:09:54.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.478+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f59e0026350 con 0x7f59f0060300 2023-11-16T14:09:54.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.686+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59f00fc150 con 0x7f59f0060300 2023-11-16T14:09:54.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.687+0000 7f59ecff9700 1 -- 172.21.15.112:0/2259564241 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59e0025300 con 0x7f59f0060300 2023-11-16T14:09:54.689 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:09:54.689 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 msgr2=0x7f59dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 0x7f59dc033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f59d80096a0 tx=0x7f59d8006b40).stop 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 msgr2=0x7f59f0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f59e0000c00 tx=0x7f59e000b040).stop 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 shutdown_connections 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f59dc031090 0x7f59dc033550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 --2- 172.21.15.112:0/2259564241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f0060300 0x7f59f0106780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 >> 172.21.15.112:0/2259564241 conn(0x7f59f00f0be0 msgr2=0x7f59f00f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 shutdown_connections 2023-11-16T14:09:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:54.689+0000 7f59f656c700 1 -- 172.21.15.112:0/2259564241 wait complete. 2023-11-16T14:09:54.692 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:09:55.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:54 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:54.687508+0000 mon.smithi061 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.112:0/2259564241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:09:56.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:55 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:54.919463+0000 mgr.smithi061.zcjido (mgr.14156) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:09:56.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:09:56.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:09:57.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 -- 172.21.15.112:0/1220062452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 msgr2=0x7f26200f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 --2- 172.21.15.112:0/1220062452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f26200f56f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f2610009fb0 tx=0x7f2610009b30).stop 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 -- 172.21.15.112:0/1220062452 shutdown_connections 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 --2- 172.21.15.112:0/1220062452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f26200f56f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 -- 172.21.15.112:0/1220062452 >> 172.21.15.112:0/1220062452 conn(0x7f26200f0a30 msgr2=0x7f26200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 -- 172.21.15.112:0/1220062452 shutdown_connections 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.137+0000 7f262491a700 1 -- 172.21.15.112:0/1220062452 wait complete. 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f262491a700 1 Processor -- start 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f262491a700 1 -- start start 2023-11-16T14:10:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f262491a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f262491a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620106b30 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f261ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f261ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44076/0 (socket says 172.21.15.112:44076) 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.138+0000 7f261ed9d700 1 -- 172.21.15.112:0/3977751771 learned_addr learned my addr 172.21.15.112:0/3977751771 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f261ed9d700 1 -- 172.21.15.112:0/3977751771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610005040 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f261ed9d700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f26100089f0 tx=0x7f261000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2610009320 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2620106d30 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2610013ec0 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2610014d30 con 0x7f26200f52d0 2023-11-16T14:10:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.139+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26201071c0 con 0x7f26200f52d0 2023-11-16T14:10:02.617 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:56.919822+0000 mgr.smithi061.zcjido (mgr.14156) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:02.617 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:57 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:09:57.353973+0000 mon.smithi061 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.112:0/3977751771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:02.617 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:09:59 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:09:58.920180+0000 mgr.smithi061.zcjido (mgr.14156) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:02.638 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:01 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:00.920561+0000 mgr.smithi061.zcjido (mgr.14156) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.140+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f26100095f0 con 0x7f26200f52d0 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.140+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620045bb0 con 0x7f26200f52d0 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.140+0000 7f26077fe700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 0x7f2608033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.140+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26100424a0 con 0x7f26200f52d0 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.140+0000 7f261e59c700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 0x7f2608033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.141+0000 7f261e59c700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 0x7f2608033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f26140096a0 tx=0x7f2614006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.145+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f261001d7f0 con 0x7f26200f52d0 2023-11-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.353+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2620108070 con 0x7f26200f52d0 2023-11-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.353+0000 7f26077fe700 1 -- 172.21.15.112:0/3977751771 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f261001d7f0 con 0x7f26200f52d0 2023-11-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 msgr2=0x7f2608033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 0x7f2608033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f26140096a0 tx=0x7f2614006b40).stop 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 msgr2=0x7f2620106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f26100089f0 tx=0x7f261000b040).stop 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 shutdown_connections 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f2608031090 0x7f2608033550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 --2- 172.21.15.112:0/3977751771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26200f52d0 0x7f2620106640 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 >> 172.21.15.112:0/3977751771 conn(0x7f26200f0a30 msgr2=0x7f26200f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 shutdown_connections 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:09:57.355+0000 7f262491a700 1 -- 172.21.15.112:0/3977751771 wait complete. 2023-11-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:03.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:03.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:04.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:03 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:02.920940+0000 mgr.smithi061.zcjido (mgr.14156) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:04.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 -- 172.21.15.112:0/3784052748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 msgr2=0x7fdabc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 --2- 172.21.15.112:0/3784052748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc0f69b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fdab0004760 tx=0x7fdab0009ad0).stop 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 -- 172.21.15.112:0/3784052748 shutdown_connections 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 --2- 172.21.15.112:0/3784052748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc0f69b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 -- 172.21.15.112:0/3784052748 >> 172.21.15.112:0/3784052748 conn(0x7fdabc0f1cd0 msgr2=0x7fdabc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 -- 172.21.15.112:0/3784052748 shutdown_connections 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.740+0000 7fdac0924700 1 -- 172.21.15.112:0/3784052748 wait complete. 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdac0924700 1 Processor -- start 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdac0924700 1 -- start start 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdac0924700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdac0924700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc106cb0 con 0x7fdabc0f6590 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdaba59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdaba59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:44082/0 (socket says 172.21.15.112:44082) 2023-11-16T14:10:04.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdaba59c700 1 -- 172.21.15.112:0/3246457230 learned_addr learned my addr 172.21.15.112:0/3246457230 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.741+0000 7fdaba59c700 1 -- 172.21.15.112:0/3246457230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009d70 con 0x7fdabc0f6590 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdaba59c700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fdab0013f10 tx=0x7fdab0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab000e040 con 0x7fdabc0f6590 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdab0007df0 con 0x7fdabc0f6590 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc106eb0 con 0x7fdabc0f6590 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab0014d60 con 0x7fdabc0f6590 2023-11-16T14:10:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.742+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc107340 con 0x7fdabc0f6590 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdab0007960 con 0x7fdabc0f6590 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc045bb0 con 0x7fdabc0f6590 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdaaaffd700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 0x7fdaa4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdab00424e0 con 0x7fdabc0f6590 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdab9d9b700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 0x7fdaa4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.743+0000 7fdab9d9b700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 0x7fdaa4033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fdaac0096a0 tx=0x7fdaac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:04.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.748+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdab0026c70 con 0x7fdabc0f6590 2023-11-16T14:10:04.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.956+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdabc1081f0 con 0x7fdabc0f6590 2023-11-16T14:10:04.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.957+0000 7fdaaaffd700 1 -- 172.21.15.112:0/3246457230 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdab001d320 con 0x7fdabc0f6590 2023-11-16T14:10:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:04.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 msgr2=0x7fdaa4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 0x7fdaa4033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fdaac0096a0 tx=0x7fdaac006b40).stop 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 msgr2=0x7fdabc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fdab0013f10 tx=0x7fdab0009ad0).stop 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 shutdown_connections 2023-11-16T14:10:04.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fdaa4031090 0x7fdaa4033550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:04.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 --2- 172.21.15.112:0/3246457230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdabc0f6590 0x7fdabc1067c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:04.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 >> 172.21.15.112:0/3246457230 conn(0x7fdabc0f1cd0 msgr2=0x7fdabc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:04.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 shutdown_connections 2023-11-16T14:10:04.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:04.958+0000 7fdac0924700 1 -- 172.21.15.112:0/3246457230 wait complete. 2023-11-16T14:10:04.962 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:05.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:05 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:04.958152+0000 mon.smithi061 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.112:0/3246457230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:06.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:06.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:06.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:06 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:04.921353+0000 mgr.smithi061.zcjido (mgr.14156) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:07.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- 172.21.15.112:0/1728050726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 msgr2=0x7fb8c00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/1728050726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c00f69b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fb8b4004760 tx=0x7fb8b4009ad0).stop 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- 172.21.15.112:0/1728050726 shutdown_connections 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/1728050726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c00f69b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- 172.21.15.112:0/1728050726 >> 172.21.15.112:0/1728050726 conn(0x7fb8c00f1cd0 msgr2=0x7fb8c00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- 172.21.15.112:0/1728050726 shutdown_connections 2023-11-16T14:10:07.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- 172.21.15.112:0/1728050726 wait complete. 2023-11-16T14:10:07.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 Processor -- start 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- start start 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.436+0000 7fb8c5a08700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0106cb0 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:39648/0 (socket says 172.21.15.112:39648) 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8beffd700 1 -- 172.21.15.112:0/2313656873 learned_addr learned my addr 172.21.15.112:0/2313656873 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8beffd700 1 -- 172.21.15.112:0/2313656873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4009d70 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8beffd700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb8b4004170 tx=0x7fb8b4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b400e040 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c0106eb0 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb8b4008ef0 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b4014c90 con 0x7fb8c00f6590 2023-11-16T14:10:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.437+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c0107340 con 0x7fb8c00f6590 2023-11-16T14:10:07.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.438+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb8b4014420 con 0x7fb8c00f6590 2023-11-16T14:10:07.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.438+0000 7fb8a77fe700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 0x7fb8a80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:07.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.438+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb8b40422e0 con 0x7fb8c00f6590 2023-11-16T14:10:07.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.438+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c0045bb0 con 0x7fb8c00f6590 2023-11-16T14:10:07.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.439+0000 7fb8be7fc700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 0x7fb8a80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:07.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.439+0000 7fb8be7fc700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 0x7fb8a80335a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8b00096a0 tx=0x7fb8b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.444+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb8b4012d00 con 0x7fb8c00f6590 2023-11-16T14:10:07.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.653+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c00fc080 con 0x7fb8c00f6590 2023-11-16T14:10:07.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.654+0000 7fb8a77fe700 1 -- 172.21.15.112:0/2313656873 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8c00fc080 con 0x7fb8c00f6590 2023-11-16T14:10:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:07.657 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:07.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 msgr2=0x7fb8a80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:07.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 0x7fb8a80335a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8b00096a0 tx=0x7fb8b0006b40).stop 2023-11-16T14:10:07.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 msgr2=0x7fb8c01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:07.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb8b4004170 tx=0x7fb8b4009ad0).stop 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 shutdown_connections 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb8a80310e0 0x7fb8a80335a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 --2- 172.21.15.112:0/2313656873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00f6590 0x7fb8c01067c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 >> 172.21.15.112:0/2313656873 conn(0x7fb8c00f1cd0 msgr2=0x7fb8c00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.655+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 shutdown_connections 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:07.656+0000 7fb8c5a08700 1 -- 172.21.15.112:0/2313656873 wait complete. 2023-11-16T14:10:07.659 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:08.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:06.921684+0000 mgr.smithi061.zcjido (mgr.14156) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:08.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:08 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:07.655472+0000 mon.smithi061 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.112:0/2313656873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:09.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:09.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 -- 172.21.15.112:0/648968852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 msgr2=0x7ff5f00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/648968852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f00f6800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff5d8009fb0 tx=0x7ff5d8009b30).stop 2023-11-16T14:10:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 -- 172.21.15.112:0/648968852 shutdown_connections 2023-11-16T14:10:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/648968852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f00f6800 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 -- 172.21.15.112:0/648968852 >> 172.21.15.112:0/648968852 conn(0x7ff5f00f1b40 msgr2=0x7ff5f00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 -- 172.21.15.112:0/648968852 shutdown_connections 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.138+0000 7ff5f49e3700 1 -- 172.21.15.112:0/648968852 wait complete. 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5f49e3700 1 Processor -- start 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5f49e3700 1 -- start start 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5f49e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5f49e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0069f00 con 0x7ff5f00f63e0 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5eed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5eed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:39660/0 (socket says 172.21.15.112:39660) 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.139+0000 7ff5eed9d700 1 -- 172.21.15.112:0/2525271127 learned_addr learned my addr 172.21.15.112:0/2525271127 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5eed9d700 1 -- 172.21.15.112:0/2525271127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8005040 con 0x7ff5f00f63e0 2023-11-16T14:10:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5eed9d700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7ff5d80089f0 tx=0x7ff5d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d8009320 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f006a0a0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff5d8013ec0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d8014d30 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.140+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f006a530 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.141+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff5d80095f0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.141+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f00591e0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.141+0000 7ff5e77fe700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 0x7ff5dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.141+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff5d80424e0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.141+0000 7ff5ee59c700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 0x7ff5dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:10.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.142+0000 7ff5ee59c700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 0x7ff5dc033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7ff5e00096a0 tx=0x7ff5e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.146+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff5d801d7f0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.356+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5f0045bb0 con 0x7ff5f00f63e0 2023-11-16T14:10:10.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.356+0000 7ff5e77fe700 1 -- 172.21.15.112:0/2525271127 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5d800fe40 con 0x7ff5f00f63e0 2023-11-16T14:10:10.360 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:10.360 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:10.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 msgr2=0x7ff5dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:10.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 0x7ff5dc033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7ff5e00096a0 tx=0x7ff5e0006b40).stop 2023-11-16T14:10:10.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 msgr2=0x7ff5f006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:10.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7ff5d80089f0 tx=0x7ff5d800b040).stop 2023-11-16T14:10:10.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 shutdown_connections 2023-11-16T14:10:10.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7ff5dc031090 0x7ff5dc033550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:10.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 --2- 172.21.15.112:0/2525271127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5f00f63e0 0x7ff5f006ba80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:10.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 >> 172.21.15.112:0/2525271127 conn(0x7ff5f00f1b40 msgr2=0x7ff5f00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:10.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 shutdown_connections 2023-11-16T14:10:10.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:10.358+0000 7ff5f49e3700 1 -- 172.21.15.112:0/2525271127 wait complete. 2023-11-16T14:10:10.364 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:10.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:10 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:08.921976+0000 mgr.smithi061.zcjido (mgr.14156) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:11.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:11 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:10.358015+0000 mon.smithi061 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.112:0/2525271127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:11.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:11.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:12.508 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:12 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:10.922262+0000 mgr.smithi061.zcjido (mgr.14156) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 -- 172.21.15.112:0/3100631069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 msgr2=0x7fb7780f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 --2- 172.21.15.112:0/3100631069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb7780f91c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fb760009fb0 tx=0x7fb760009b30).stop 2023-11-16T14:10:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 -- 172.21.15.112:0/3100631069 shutdown_connections 2023-11-16T14:10:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 --2- 172.21.15.112:0/3100631069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb7780f91c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 -- 172.21.15.112:0/3100631069 >> 172.21.15.112:0/3100631069 conn(0x7fb7780f0be0 msgr2=0x7fb7780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 -- 172.21.15.112:0/3100631069 shutdown_connections 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.843+0000 7fb77decd700 1 -- 172.21.15.112:0/3100631069 wait complete. 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb77decd700 1 Processor -- start 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb77decd700 1 -- start start 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb77decd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb77decd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778106c70 con 0x7fb778060300 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb7777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb7777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:39686/0 (socket says 172.21.15.112:39686) 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb7777fe700 1 -- 172.21.15.112:0/1295648069 learned_addr learned my addr 172.21.15.112:0/1295648069 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.844+0000 7fb7777fe700 1 -- 172.21.15.112:0/1295648069 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760005040 con 0x7fb778060300 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb7777fe700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fb760009f80 tx=0x7fb76000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb760009320 con 0x7fb778060300 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb778106e70 con 0x7fb778060300 2023-11-16T14:10:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb760013680 con 0x7fb778060300 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76001eda0 con 0x7fb778060300 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.845+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb778107300 con 0x7fb778060300 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb760014420 con 0x7fb778060300 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb76ffff700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 0x7fb764033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb7600423c0 con 0x7fb778060300 2023-11-16T14:10:12.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780591e0 con 0x7fb778060300 2023-11-16T14:10:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb776ffd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 0x7fb764033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.846+0000 7fb776ffd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 0x7fb764033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb7680096a0 tx=0x7fb768006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:12.851+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb76001c910 con 0x7fb778060300 2023-11-16T14:10:13.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.061+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb778045bb0 con 0x7fb778060300 2023-11-16T14:10:13.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.061+0000 7fb76ffff700 1 -- 172.21.15.112:0/1295648069 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb76001c320 con 0x7fb778060300 2023-11-16T14:10:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:13.064 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:13.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 msgr2=0x7fb764033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:13.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 0x7fb764033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb7680096a0 tx=0x7fb768006b40).stop 2023-11-16T14:10:13.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 msgr2=0x7fb778106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:13.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fb760009f80 tx=0x7fb76000b040).stop 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 shutdown_connections 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb764031090 0x7fb764033550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 --2- 172.21.15.112:0/1295648069 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb778060300 0x7fb778106780 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 >> 172.21.15.112:0/1295648069 conn(0x7fb7780f0be0 msgr2=0x7fb7780f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 shutdown_connections 2023-11-16T14:10:13.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:13.063+0000 7fb77decd700 1 -- 172.21.15.112:0/1295648069 wait complete. 2023-11-16T14:10:13.068 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:12.922632+0000 mgr.smithi061.zcjido (mgr.14156) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:14.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:13 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:13.062895+0000 mon.smithi061 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.112:0/1295648069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:14.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:14.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:15.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 -- 172.21.15.112:0/1141587959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 msgr2=0x7f692c0f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 --2- 172.21.15.112:0/1141587959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c0f4c70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f6924009fb0 tx=0x7f6924009b30).stop 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 -- 172.21.15.112:0/1141587959 shutdown_connections 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 --2- 172.21.15.112:0/1141587959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c0f4c70 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 -- 172.21.15.112:0/1141587959 >> 172.21.15.112:0/1141587959 conn(0x7f692c0f0390 msgr2=0x7f692c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 -- 172.21.15.112:0/1141587959 shutdown_connections 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.501+0000 7f693120e700 1 -- 172.21.15.112:0/1141587959 wait complete. 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f693120e700 1 Processor -- start 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f693120e700 1 -- start start 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f693120e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f693120e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c106c90 con 0x7f692c0f4850 2023-11-16T14:10:15.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:15.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46208/0 (socket says 172.21.15.112:46208) 2023-11-16T14:10:15.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f692ad9d700 1 -- 172.21.15.112:0/1613396296 learned_addr learned my addr 172.21.15.112:0/1613396296 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f692ad9d700 1 -- 172.21.15.112:0/1613396296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924005040 con 0x7f692c0f4850 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.502+0000 7f692ad9d700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f69240089f0 tx=0x7f692400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6924009320 con 0x7f692c0f4850 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6924013ec0 con 0x7f692c0f4850 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c106e90 con 0x7f692c0f4850 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6924014d30 con 0x7f692c0f4850 2023-11-16T14:10:15.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c107320 con 0x7f692c0f4850 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f69240095f0 con 0x7f692c0f4850 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.503+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c045bb0 con 0x7f692c0f4850 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.504+0000 7f69237fe700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 0x7f6918033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.504+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f69240424a0 con 0x7f692c0f4850 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.504+0000 7f692a59c700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 0x7f6918033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:15.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.504+0000 7f692a59c700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 0x7f6918033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f69140096a0 tx=0x7f6914006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:15.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.509+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f692401d7f0 con 0x7f692c0f4850 2023-11-16T14:10:15.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.717+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f692c1081d0 con 0x7f692c0f4850 2023-11-16T14:10:15.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.718+0000 7f69237fe700 1 -- 172.21.15.112:0/1613396296 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f692401d7f0 con 0x7f692c0f4850 2023-11-16T14:10:15.722 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:15.722 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 msgr2=0x7f6918033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 0x7f6918033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f69140096a0 tx=0x7f6914006b40).stop 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 msgr2=0x7f692c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f69240089f0 tx=0x7f692400b040).stop 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 shutdown_connections 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f6918031090 0x7f6918033550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:15.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 --2- 172.21.15.112:0/1613396296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f692c0f4850 0x7f692c1067a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:15.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 >> 172.21.15.112:0/1613396296 conn(0x7f692c0f0390 msgr2=0x7f692c0f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:15.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 shutdown_connections 2023-11-16T14:10:15.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:15.720+0000 7f693120e700 1 -- 172.21.15.112:0/1613396296 wait complete. 2023-11-16T14:10:15.724 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:14.922964+0000 mgr.smithi061.zcjido (mgr.14156) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:16.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:15 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:15.719467+0000 mon.smithi061 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.112:0/1613396296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:17.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:17.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:18.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- 172.21.15.112:0/4103549882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 msgr2=0x7f799c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:18.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 --2- 172.21.15.112:0/4103549882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c0f58a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f798c009fb0 tx=0x7f798c009b30).stop 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- 172.21.15.112:0/4103549882 shutdown_connections 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 --2- 172.21.15.112:0/4103549882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c0f58a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- 172.21.15.112:0/4103549882 >> 172.21.15.112:0/4103549882 conn(0x7f799c0f0be0 msgr2=0x7f799c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- 172.21.15.112:0/4103549882 shutdown_connections 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- 172.21.15.112:0/4103549882 wait complete. 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 Processor -- start 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.168+0000 7f79a0845700 1 -- start start 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f79a0845700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f79a0845700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c106ce0 con 0x7f799c0f5480 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f799a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f799a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46232/0 (socket says 172.21.15.112:46232) 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f799a59c700 1 -- 172.21.15.112:0/1475660580 learned_addr learned my addr 172.21.15.112:0/1475660580 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f799a59c700 1 -- 172.21.15.112:0/1475660580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c005040 con 0x7f799c0f5480 2023-11-16T14:10:18.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f799a59c700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f798c0089f0 tx=0x7f798c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798c009320 con 0x7f799c0f5480 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c106ee0 con 0x7f799c0f5480 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f798c013ec0 con 0x7f799c0f5480 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798c014d30 con 0x7f799c0f5480 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.169+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c107370 con 0x7f799c0f5480 2023-11-16T14:10:18.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.170+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f798c0095f0 con 0x7f799c0f5480 2023-11-16T14:10:18.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.170+0000 7f798affd700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 0x7f7984033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:18.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.170+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c045bb0 con 0x7f799c0f5480 2023-11-16T14:10:18.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.170+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f798c0424e0 con 0x7f799c0f5480 2023-11-16T14:10:18.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.171+0000 7f7999d9b700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 0x7f7984033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:18.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.171+0000 7f7999d9b700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 0x7f7984033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:18.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.175+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f798c01d7f0 con 0x7f799c0f5480 2023-11-16T14:10:18.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:17 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:16.923396+0000 mgr.smithi061.zcjido (mgr.14156) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:18.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.385+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f799c108190 con 0x7f799c0f5480 2023-11-16T14:10:18.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.386+0000 7f798affd700 1 -- 172.21.15.112:0/1475660580 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f798c02c090 con 0x7f799c0f5480 2023-11-16T14:10:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:18.390 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 msgr2=0x7f7984033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 0x7f7984033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).stop 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 msgr2=0x7f799c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f798c0089f0 tx=0x7f798c00b040).stop 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 shutdown_connections 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f7984031090 0x7f7984033550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 --2- 172.21.15.112:0/1475660580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f799c0f5480 0x7f799c1067f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 >> 172.21.15.112:0/1475660580 conn(0x7f799c0f0be0 msgr2=0x7f799c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:18.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 shutdown_connections 2023-11-16T14:10:18.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:18.388+0000 7f79a0845700 1 -- 172.21.15.112:0/1475660580 wait complete. 2023-11-16T14:10:18.392 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:19.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:18 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:18.387753+0000 mon.smithi061 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.112:0/1475660580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:19.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:19.752 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:20.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:19 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:18.923800+0000 mgr.smithi061.zcjido (mgr.14156) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:20.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/1937888716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 msgr2=0x7f3fc00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:20.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/1937888716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc00f9160 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f3fb0009fb0 tx=0x7f3fb0009b30).stop 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/1937888716 shutdown_connections 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/1937888716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc00f9160 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/1937888716 >> 172.21.15.112:0/1937888716 conn(0x7f3fc00f0be0 msgr2=0x7f3fc00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/1937888716 shutdown_connections 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/1937888716 wait complete. 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.860+0000 7f3fc5bf2700 1 Processor -- start 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fc5bf2700 1 -- start start 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fc5bf2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fc5bf2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0106c40 con 0x7f3fc0060360 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46250/0 (socket says 172.21.15.112:46250) 2023-11-16T14:10:20.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fbf7fe700 1 -- 172.21.15.112:0/2405371957 learned_addr learned my addr 172.21.15.112:0/2405371957 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fbf7fe700 1 -- 172.21.15.112:0/2405371957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0005040 con 0x7f3fc0060360 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.861+0000 7f3fbf7fe700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f3fb0009f80 tx=0x7f3fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb0009320 con 0x7f3fc0060360 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc0106e40 con 0x7f3fc0060360 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fb0013680 con 0x7f3fc0060360 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb001eda0 con 0x7f3fc0060360 2023-11-16T14:10:20.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc01072d0 con 0x7f3fc0060360 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.862+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fb001e420 con 0x7f3fc0060360 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.863+0000 7f3fb7fff700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.863+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc0045bb0 con 0x7f3fc0060360 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.863+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fb0042440 con 0x7f3fc0060360 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.863+0000 7f3fbeffd700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:20.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.863+0000 7f3fbeffd700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f3fa80096a0 tx=0x7f3fa8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:20.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:20.868+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fb001d380 con 0x7f3fc0060360 2023-11-16T14:10:21.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.076+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fc0108180 con 0x7f3fc0060360 2023-11-16T14:10:21.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.076+0000 7f3fb7fff700 1 -- 172.21.15.112:0/2405371957 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fb002c2d0 con 0x7f3fc0060360 2023-11-16T14:10:21.080 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:21.080 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 msgr2=0x7f3fac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f3fa80096a0 tx=0x7f3fa8006b40).stop 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 msgr2=0x7f3fc0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f3fb0009f80 tx=0x7f3fb000b040).stop 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 shutdown_connections 2023-11-16T14:10:21.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7f3fac031090 0x7f3fac033550 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:21.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 --2- 172.21.15.112:0/2405371957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fc0060360 0x7f3fc0106750 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:21.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 >> 172.21.15.112:0/2405371957 conn(0x7f3fc00f0be0 msgr2=0x7f3fc00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:21.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 shutdown_connections 2023-11-16T14:10:21.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:21.078+0000 7f3fc5bf2700 1 -- 172.21.15.112:0/2405371957 wait complete. 2023-11-16T14:10:21.082 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:20.924126+0000 mgr.smithi061.zcjido (mgr.14156) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:22.258 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:21 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: audit 2023-11-16T14:10:21.078193+0000 mon.smithi061 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.112:0/2405371957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T14:10:22.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T14:10:22.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 -- 172.21.15.112:0/3091781025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 msgr2=0x7fb5380f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 --2- 172.21.15.112:0/3091781025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f9200 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb528009fb0 tx=0x7fb528009b30).stop 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 -- 172.21.15.112:0/3091781025 shutdown_connections 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 --2- 172.21.15.112:0/3091781025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f9200 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 -- 172.21.15.112:0/3091781025 >> 172.21.15.112:0/3091781025 conn(0x7fb5380f0be0 msgr2=0x7fb5380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 -- 172.21.15.112:0/3091781025 shutdown_connections 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.545+0000 7fb5402d1700 1 -- 172.21.15.112:0/3091781025 wait complete. 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb5402d1700 1 Processor -- start 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb5402d1700 1 -- start start 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb5402d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:23.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb5402d1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380f5100 con 0x7fb5380602c0 2023-11-16T14:10:23.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb53e06d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:23.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb53e06d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.112:46274/0 (socket says 172.21.15.112:46274) 2023-11-16T14:10:23.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.546+0000 7fb53e06d700 1 -- 172.21.15.112:0/2234251529 learned_addr learned my addr 172.21.15.112:0/2234251529 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-11-16T14:10:23.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb53e06d700 1 -- 172.21.15.112:0/2234251529 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528005040 con 0x7fb5380602c0 2023-11-16T14:10:23.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb53e06d700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fb528013fd0 tx=0x7fb528009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb528018070 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5380f52a0 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb528007b10 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52801c3c0 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.547+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380f5730 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.548+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb52801c640 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.548+0000 7fb52effd700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 0x7fb524033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.548+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb5280429e0 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.548+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5380591e0 con 0x7fb5380602c0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.548+0000 7fb53d86c700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 0x7fb524033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T14:10:23.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.549+0000 7fb53d86c700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 0x7fb524033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb5340096a0 tx=0x7fb534006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T14:10:23.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.553+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb52801b550 con 0x7fb5380602c0 2023-11-16T14:10:23.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.760+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb538045bb0 con 0x7fb5380602c0 2023-11-16T14:10:23.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.761+0000 7fb52effd700 1 -- 172.21.15.112:0/2234251529 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb538045bb0 con 0x7fb5380602c0 2023-11-16T14:10:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-11-16T14:10:23.764 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"71d22a3a-8488-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T14:00:47.298126Z","created":"2023-11-16T14:00:47.298126Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T14:10:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.762+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 msgr2=0x7fb524033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.762+0000 7fb5402d1700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 0x7fb524033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb5340096a0 tx=0x7fb534006b40).stop 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.762+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 msgr2=0x7fb5380f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.762+0000 7fb5402d1700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fb528013fd0 tx=0x7fb528009b30).stop 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 shutdown_connections 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:6800/4132389775,v1:172.21.15.61:6801/4132389775] conn(0x7fb524031090 0x7fb524033550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 --2- 172.21.15.112:0/2234251529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5380602c0 0x7fb5380f6c80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 >> 172.21.15.112:0/2234251529 conn(0x7fb5380f0be0 msgr2=0x7fb5380f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 shutdown_connections 2023-11-16T14:10:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-11-16T14:10:23.763+0000 7fb5402d1700 1 -- 172.21.15.112:0/2234251529 wait complete. 2023-11-16T14:10:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2023-11-16T14:10:24.008 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:23 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: cluster 2023-11-16T14:10:22.924511+0000 mgr.smithi061.zcjido (mgr.14156) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T14:10:24.119 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-16T14:10:24.121 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-16T14:10:24.121 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T14:10:24.147 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T14:10:24.165 INFO:tasks.cephadm:Stopping all daemons... 2023-11-16T14:10:24.165 INFO:tasks.cephadm.mon.smithi061:Stopping mon.smithi061... 2023-11-16T14:10:24.165 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061 2023-11-16T14:10:24.509 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:24 smithi061 systemd[1]: Stopping Ceph mon.smithi061 for 71d22a3a-8488-11ee-8dbe-212e2dc638e7... 2023-11-16T14:10:24.510 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:24 smithi061 podman[117916]: Error: no container with name or ID "ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon.smithi061" found: no such container 2023-11-16T14:10:24.510 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:10:24.422+0000 7f187af6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi061 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-16T14:10:24.510 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Nov 16 14:10:24 smithi061 ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7-mon-smithi061[111911]: debug 2023-11-16T14:10:24.422+0000 7f187af6a700 -1 mon.smithi061@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T14:10:24.705 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi061.service' 2023-11-16T14:10:24.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T14:10:24.749 INFO:tasks.cephadm.mon.smithi061:Stopped mon.smithi061 2023-11-16T14:10:24.749 INFO:tasks.cephadm.mon.smithi112:Stopping mon.smithi112... 2023-11-16T14:10:24.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112 2023-11-16T14:10:24.783 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112.service: Unit ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112.service not loaded. 2023-11-16T14:10:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-16T14:10:24.785 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi112" Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112' 2023-11-16T14:10:24.786 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-16T14:10:24.788 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi061/crash 2023-11-16T14:10:24.789 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T14:10:24.823 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi112/crash 2023-11-16T14:10:24.824 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T14:10:24.862 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T14:10:24.862 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2023-11-16T14:10:24.863 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-16T14:10:24.863 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-16T14:10:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:grep: /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-16T14:10:24.901 INFO:tasks.cephadm:Compressing logs... 2023-11-16T14:10:24.901 DEBUG:teuthology.orchestra.run.smithi061:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T14:10:24.943 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T14:10:24.969 INFO:teuthology.orchestra.run.smithi061.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-16T14:10:24.969 INFO:teuthology.orchestra.run.smithi061.stderr:: No such file or directory 2023-11-16T14:10:24.969 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-16T14:10:24.969 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mon.smithi061.log 2023-11-16T14:10:24.970 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi061.zcjido.log 2023-11-16T14:10:24.970 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mon.smithi061.log: /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi061.zcjido.log: 90.2% -- replaced with /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-16T14:10:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-16T14:10:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:: No such file or directory 2023-11-16T14:10:24.974 INFO:teuthology.orchestra.run.smithi061.stderr: 92.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-16T14:10:24.975 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cephadm.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-16T14:10:24.976 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-11-16T14:10:24.976 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m0.015s 2023-11-16T14:10:24.976 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m0.014s 2023-11-16T14:10:24.976 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.010s 2023-11-16T14:10:25.043 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi061.zcjido.log: file size changed while zipping 2023-11-16T14:10:25.044 INFO:teuthology.orchestra.run.smithi061.stderr: 92.8% -- replaced with /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi061.zcjido.log.gz 2023-11-16T14:10:25.117 INFO:teuthology.orchestra.run.smithi061.stderr: 90.4% -- replaced with /var/log/ceph/71d22a3a-8488-11ee-8dbe-212e2dc638e7/ceph-mon.smithi061.log.gz 2023-11-16T14:10:25.119 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-11-16T14:10:25.119 INFO:teuthology.orchestra.run.smithi061.stderr:real 0m0.160s 2023-11-16T14:10:25.119 INFO:teuthology.orchestra.run.smithi061.stderr:user 0m0.232s 2023-11-16T14:10:25.119 INFO:teuthology.orchestra.run.smithi061.stderr:sys 0m0.019s 2023-11-16T14:10:25.119 INFO:tasks.cephadm:Archiving logs... 2023-11-16T14:10:25.119 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi061/log 2023-11-16T14:10:25.120 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T14:10:25.233 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi112/log 2023-11-16T14:10:25.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T14:10:25.263 INFO:tasks.cephadm:Removing cluster... 2023-11-16T14:10:25.263 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 --force 2023-11-16T14:11:31.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 71d22a3a-8488-11ee-8dbe-212e2dc638e7 --force 2023-11-16T14:11:32.442 INFO:tasks.cephadm:Removing cephadm ... 2023-11-16T14:11:32.443 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T14:11:32.464 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T14:11:32.483 INFO:tasks.cephadm:Teardown complete 2023-11-16T14:11:32.483 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112' 2023-11-16T14:11:32.562 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ffc2ae3e55d14e42b6b0221f99add2f2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae6ce92c0812ba06c9c623ce72df4f3fc59080b4/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112' 2023-11-16T14:11:32.567 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-16T14:11:32.576 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T14:11:32.598 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T14:11:32.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T14:11:32.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T14:11:32.694 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T14:11:32.703 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T14:11:32.704 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T14:11:32.707 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T14:11:32.723 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-11-16T14:11:32.724 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +512us[ +521us] +/- 78ms 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -812us[ -803us] +/- 62ms 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +736us[ +745us] +/- 87ms 2023-11-16T14:11:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T14:11:32.729 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T14:11:32.729 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-11-16T14:11:32.730 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +472us[ +472us] +/- 78ms 2023-11-16T14:11:32.730 INFO:teuthology.orchestra.run.smithi061.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -848us[ -813us] +/- 62ms 2023-11-16T14:11:32.730 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +703us[ +703us] +/- 87ms 2023-11-16T14:11:32.730 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T14:11:32.730 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T14:11:32.740 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T14:11:32.741 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T14:11:32.750 DEBUG:teuthology.orchestra.run.smithi061:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T14:11:32.864 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T14:11:32.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T14:11:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T14:11:32.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T14:11:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T14:11:33.022 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T14:11:33.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T14:11:33.079 INFO:teuthology.task.internal:Duration was 1401.650621 seconds 2023-11-16T14:11:33.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T14:11:33.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T14:11:33.088 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T14:11:33.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T14:11:33.137 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T14:11:33.138 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T14:11:33.271 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T14:11:33.272 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2023-11-16T14:11:33.272 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T14:11:33.294 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2023-11-16T14:11:33.295 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T14:11:33.317 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T14:11:33.318 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T14:11:33.338 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T14:11:33.402 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T14:11:33.402 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T14:11:33.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T14:11:33.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T14:11:33.532 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T14:11:33.532 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T14:11:33.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T14:11:33.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T14:11:33.598 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T14:11:33.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T14:11:33.627 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2023-11-16T14:11:33.654 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2023-11-16T14:11:33.677 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T14:11:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T14:11:33.718 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T14:11:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T14:11:33.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T14:11:33.747 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T14:11:33.747 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi061 2023-11-16T14:11:33.748 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T14:11:33.800 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812/remote/smithi112 2023-11-16T14:11:33.800 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T14:11:33.839 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T14:11:33.839 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T14:11:33.841 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T14:11:33.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T14:11:33.905 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T14:11:33.905 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T14:11:33.922 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T14:11:33.922 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T14:11:33.925 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T14:11:33.941 INFO:teuthology.orchestra.run.smithi061.stdout: 526081 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 14:11 /home/ubuntu/cephtest 2023-11-16T14:11:33.948 INFO:teuthology.orchestra.run.smithi112.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 14:11 /home/ubuntu/cephtest 2023-11-16T14:11:33.951 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T14:11:33.964 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T14:11:33.994 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T14:11:34.035 DEBUG:teuthology.nuke:shortname: smithi061 2023-11-16T14:11:34.035 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T14:11:34.064 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 13:39:23.612701', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJvIhhtaHs2kljBp2KydiS8aDaamE2L/NVmrcaJ0oiamX5Jo7A9fHWw4xvlHfoe7F2av3aUiEFGmN2C1i7wdZU='} 2023-11-16T14:11:34.069 DEBUG:teuthology.nuke:shortname: smithi112 2023-11-16T14:11:34.069 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T14:11:34.096 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-11-16_12:07:54-orch:cephadm-wip-63555-quincy-distro-default-smithi/7460812', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 13:39:23.614296', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbouQQgrOaglJ3snHsZZY6SjfVoJ5UJPVnHLe0Hg+rXqG/G/VNiDd/XEiDUw9grCYbfoaz8E9i+KhWtnkjqlwI='} 2023-11-16T14:11:34.125 INFO:teuthology.orchestra.console.smithi061:Power off 2023-11-16T14:11:34.126 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T14:11:34.156 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off 2023-11-16T14:11:34.156 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:34.173 INFO:teuthology.orchestra.console.smithi112:Power off 2023-11-16T14:11:34.174 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T14:11:34.192 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-11-16T14:11:34.207 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2023-11-16T14:11:34.207 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:34.249 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-11-16T14:11:38.194 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:38.250 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:38.309 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-11-16T14:11:38.366 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-11-16T14:11:42.314 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:42.367 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:42.424 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-11-16T14:11:42.479 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-11-16T14:11:46.427 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:46.479 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T14:11:46.537 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off 2023-11-16T14:11:46.593 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2023-11-16T14:11:46.639 INFO:teuthology.orchestra.console.smithi061:Power off completed 2023-11-16T14:11:46.695 INFO:teuthology.orchestra.console.smithi112:Power off completed 2023-11-16T14:11:46.800 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1401.6506214141846 failure_reason: 'Command failed on smithi112 with status 5: ''sudo systemctl stop ceph-71d22a3a-8488-11ee-8dbe-212e2dc638e7@mon.smithi112''' flavor: default owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ffc2ae3e55d14e42b6b0221f99add2f2 status: fail success: false 2023-11-16T14:11:46.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T14:11:46.878 INFO:teuthology.run:FAIL